eval

7 stories

Stories

Patterns for Building Cybersecurity Evals
A sandboxed target, inputs that influence task difficulty, tools, and a grader.
Eugene YanJun 21, 202618 min read
Product Evals in Three Simple Steps
Label some data, align LLM-evaluators, and run the eval harness with each change.
Eugene YanNov 23, 20259 min read
Evaluating Long-Context Question & Answer Systems
Evaluation metrics, how to build eval datasets, eval methodology, and a review of several benchmarks.
Eugene YanJun 22, 202526 min read
An LLM-as-Judge Won't Save The Product—Fixing Your Process Will
Applying the scientific method, building via eval-driven development, and monitoring AI output.
Eugene YanApr 20, 20255 min read
AlignEval: Building an App to Make Evals Easy, Fun, and Automated
Look at and label your data, build and evaluate your LLM-evaluator, and optimize it against your labels.
Eugene YanOct 27, 202412 min read
Weights & Biases LLM-Evaluator Hackathon - Hackathon Judge
Being a human judge at the Weights & Biases LLM-as-a-Judge Hackathon
Eugene YanSep 22, 20242 min read
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
Use cases, techniques, alignment, finetuning, and critiques against LLM-evaluators.
Eugene YanAug 18, 202445 min read