Do testing instructions improve coding agent correctness? What 30 conditions on one Zstd task measured
In a 30-condition eval, telling Codex to use TDD cut fully correct Zstd decoders by 5 to 7 points. No testing instruction clearly beat none.
This article presents Dan Luu's 30-condition evaluation of testing instructions' impact on coding agent correctness, focusing on OpenAI Codex's performance in writing Zstandard decoders in Rust.
What happened
Dan Luu evaluated 30 testing instruction conditions for coding agents, measuring their ability to produce correct Zstandard decoders in Rust. The study found no clear improvement from testing instructions, with TDD actually reducing correctness and some methods like Hegel increasing costs without benefits.
Why it matters
The findings challenge assumptions about testing's effectiveness for AI coding tools, highlighting risks of agent-written tests and informing best practices for AI-assisted software development.
- Testing instructions may not improve coding agent correctness
- TDD can reduce decoder quality despite increased test volume
- Property-based testing shows limited but measurable value
- Formal methods require careful implementation to avoid cost inefficiencies
- Agent-written tests often lack substantive validation
The bigger picture
The study suggests current AI coding tools may not benefit significantly from testing instructions, prompting reevaluation of testing strategies and emphasizing the need for external test suites.
Luu's evaluation found no conclusive evidence that testing instructions improve coding agent correctness, with results suggesting potential risks in relying on agent-generated tests for critical software components.
Technical details
- Zstandard decoder implementation in Rust
- Test-driven development (TDD) workflow
- Property-based testing with proptest
- Formal verification tools (Verus, TLA+, Lean 4)
- Cost measurement in USD per run
FAQ
Did test-driven development improve coding agent performance?
No. TDD reduced correct decoder rates by 5-7 points compared to no instruction, with agents writing more tests but fewer correct implementations.
What was the most effective testing method?
Luu's five-bullet skill achieved the highest mean correctness (78.8%), but results were within statistical noise and not significantly better than other methods.
Why do agent-written tests often fail?
Agents frequently create vacuous tests that don't validate actual functionality, with 5.16 assertions per 25.00 value-printing statements in one study.
How do formal methods perform in this context?
Formal methods like Verus and TLA+ showed limited effectiveness, with high costs and minimal correctness improvements compared to simpler approaches.
What are the implications for AI coding tools?
The study suggests current AI tools may not benefit significantly from testing instructions, emphasizing the need for external test suites and caution against over-reliance on agent-generated tests.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by Stackness Blog on Oct 10, 2026.
Comments
Sign in to join the discussion.