AI Agent Evals: The Judge Is The Part Nobody Measures
An LLM judge scoring the same agent output three times will often hand back three different answers, and almost no guide to agent evaluation asks whether the judge agrees with itself or what it costs to run on everything rather than a sample.
Maximem Synap Blog18 min read