Know a great AI tool or AI news source? Submit it to AiByts

Online Evals

6 stories

Stories

  • What LLM-as-a-Judge Costs in Production, and How to Cut It

    Online evals with an LLM judge cost judge tokens plus platform fees, and both scale with traffic. Here is the formula, a worked example, and the levers that cut cost without hiding failures: filtering, sampling, smaller judges, batching, and code checks.

    Agnost AI Blog5 min read
  • How to Set Up Online Evals for AI Agents in Production

    A practical setup for online evals: capture the right evidence, pick the unit, write narrow judge rubrics, choose sampling, calibrate against human labels, and route results to alerts and fixes. Includes a rubric template and defaults.

    Agnost AI Blog7 min read
  • Online Evals vs Guardrails vs Monitoring: Which One Catches What

    Guardrails run inline and block bad output before users see it. Online evals judge production traffic afterwards to find quality failures. Monitoring tracks system health. Here is what each catches, what each costs, and how to combine them for an AI agent.

    Agnost AI Blog5 min read
  • What Are Online Evals? Online vs Offline Evaluation for AI Agents

    Online evals score an AI agent's real production traffic after it happens, using LLM judges, code checks, or user signals. Offline evals score a fixed test set before release. Here is how they differ, what each catches, and how to run both.

    Agnost AI Blog8 min read
  • Your Online Eval Failed. Now What? From Flag to Verified Fix

    An online eval flag is the start of the work, not the end. Here is the path from a failing production eval to a confirmed root cause, a reviewed fix, a regression test, and a measured before-and-after, with the mistakes that break each step.

    Agnost AI Blog6 min read

Showing 6 of 6 stories

You're all caught up.

AiByts Weekly Brief

Get the top AI news and tools straight to your inbox every week.

Weekly · AI news

Join 15,000+ AI enthusiasts