AI Agent Evaluation: A Practical Guide with a Working Application
An end-to-end guide to deterministic checks, retrieval metrics, human review, LLM judges, and regression testing, with practical examples from a local multi-agent RAG application.
41 stories
An end-to-end guide to deterministic checks, retrieval metrics, human review, LLM judges, and regression testing, with practical examples from a local multi-agent RAG application.
A voice agent that negotiates freight rates, ten scripted attacks, and the difference between “never crossed the limit” and “never gave the limit away”
AI agents guess when a question is vague. Two things help: write down your rules first, then add a separate check that asks.Continue reading on Towards AI »
A year ago I wrote about building a tiny AI agent that uses MCP tools. It’s still the most-read thing I’ve published.
Learn the full AI agent stack: LLM, context, tools, memory, orchestration, runtime, guardrails, and observability. A clear guide to…Continue reading on Towards AI »
Your framework is the brain. Redis is the hands. Here’s when the hands are overkill.Continue reading on Towards AI »
For roughly the last ten years, I have participated in hiring students and full-time employees at a large corporation. I was one of several people involved in hiring for my unit. The process changed over time, but its general shape will probably sound familiar.
LAI #145: What Makes an Agent Reliable? Plus, persistent AI identity, decisions without decoding, causal inference, Kubernetes triage, and RAG claim verification. Good morning, AI enthusiasts! AI …
OpenAI shelved GPT-6.1 Astra after it overstepped its scope and misreported its work, then launched always-on Dots agents a day later.Continue reading on Towards AI »
Compare an AI video agent with a video generation API and a human editor. Three questions show which video tool fits your workflow.
AgenticSystemCore: a folder of Markdown texts that people and agents can use in multiple ways Existing technologies for future simpler combined understanding and use An open format that turns plain …
Signing a handle stops forgery. It does not stop theft. Here are the four checks every MCP server now needs.
Stopping the model is only the beginning. The real test is whether everything it set in motion stops with it.
Prompt Engineering vs Context Engineering. A Hands-on Cortex AI Walkthrough An example demonstrating how the same prompt and instruction return different severity ratings depending on how much …
Stop Paying Your LLM to Flip Coins 9 design patterns for System One models like Jev and Laya, and how they make AI agents faster, cheaper, and safer Here’s an uncomfortable exercise. Open your …
Agents Aren’t Chatbots With Extra Steps If you are finding it difficult to pin down what an “AI agent” actually is, some of that confusion is understandable. Two systems can look much the same …
Your AI Agent Will Obey Anyone. Guardrails Are How You Stop It. A guide to the attacks that break LLMs and AI agents, and the guardrails that stop each one. This guide assumes you know nothing about …
Triggers, identity, retries, and the reminder tool, including what a “completed” run does and doesn’t guarantee.Continue reading on Towards AI »
Showing 36 of 41 stories
Load more storiesGet the top AI news and tools straight to your inbox every week.
Weekly · AI news from Towards AI
Join 15,000+ AI enthusiasts