Benchmark Contamination: How to Seal Your Agent Evals
Benchmark contamination turns AI eval scores into recall tests. Learn how leaks happen, how to detect them, and the Sealed Suite framework for agent evals.
15 stories
Benchmark contamination turns AI eval scores into recall tests. Learn how leaks happen, how to detect them, and the Sealed Suite framework for agent evals.
Online evals with an LLM judge cost judge tokens plus platform fees, and both scale with traffic. Here is the formula, a worked example, and the levers that cut cost without hiding failures: filtering, sampling, smaller judges, batching, and code checks.
A practical setup for online evals: capture the right evidence, pick the unit, write narrow judge rubrics, choose sampling, calibrate against human labels, and route results to alerts and fixes. Includes a rubric template and defaults.
Guardrails run inline and block bad output before users see it. Online evals judge production traffic afterwards to find quality failures. Monitoring tracks system health. Here is what each catches, what each costs, and how to combine them for an AI agent.
Span and trace evals judge one model call or one request. Conversation-level evals judge the whole multi-turn session. Here is which agent failures each unit can and cannot see, and how to run both without double-counting.
Online evals score an AI agent's real production traffic after it happens, using LLM judges, code checks, or user signals. Offline evals score a fixed test set before release. Here is how they differ, what each catches, and how to run both.
An online eval flag is the start of the work, not the end. Here is the path from a failing production eval to a confirmed root cause, a reviewed fix, a regression test, and a measured before-and-after, with the mistakes that break each step.
A wrong answer that raises no error is a mismatch between what the agent said and something you can check: a tool result, a retrieved source, a system of record, or what the user did next.
A practical guide to monitoring production AI agents for wrong answers, silent failures, bad tool use, and unresolved user goals—not just exceptions and latency.
Agnost's first specialized model improved task success by 22.9% while reducing median latency by 90.2% and cost by 94.5% on 780 held-out production traces.
Define verified agent outcomes, eligible tasks, observation windows, and unknowns. Avoid counting fluent replies, HTTP 200s, or unfinished work as success.
Evaluate a model-based failure detector against human labels, healthy controls, and missing-evidence cases before trusting its alerts or scores.
Separate schema validation from factual, authorization, and execution checks. Catch well-formed agent outputs that target the wrong object or claim the wrong state.
Calculate agent cost using verified outcomes, retries, tool fees, and review overhead. Includes an illustrative comparison and a reusable measurement contract.
Detect source contradictions by matching claims to retrieved evidence, scope, and policy versions. Separate grounding errors from retrieval failures.
Showing 15 of 15 stories
You're all caught up.
Get the top AI news and tools straight to your inbox every week.
Weekly · AI news
Join 15,000+ AI enthusiasts