FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experience.
The article introduces FDE-Bench, a benchmark designed to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization in real-world business scenarios.
What happened
The article presents FDE-Bench, a new benchmark for evaluating agents on end-to-end delivery tasks that involve clarifying underspecified business requests and iteratively optimizing solutions. The benchmark is based on real customer projects and includes 49 cases across different domains. Agents are tested on their ability to elicit requirements and produce solutions that meet customer acceptance criteria.
Why it matters
This benchmark is important for AI developers and technology professionals as it addresses the challenge of handling real-world business requests that are often underspecified. It highlights the gap between current agent capabilities and the requirements of frontline engineers in delivering solutions that meet customer needs.
- Real-world business tasks often require interactive clarification and iterative optimization.
- FDE-Bench provides a new standard for evaluating AI agents in complex, underspecified scenarios.
- Current AI agents have significant limitations in handling end-to-end delivery tasks without explicit requirements.
The bigger picture
The results indicate that current agents struggle with end-to-end delivery on underspecified tasks, highlighting the need for improved methods in requirement elicitation and iterative optimization. This benchmark can guide future research and development in creating more effective AI systems for real-world business applications.
FDE-Bench is introduced as a benchmark to evaluate AI agents on end-to-end delivery tasks involving underspecified business requests. The results show that current agents fall short of FDE-level performance, underscoring the need for further research in this area.
Technical details
- FDE-Bench: A benchmark for end-to-end delivery tasks requiring interactive requirement clarification and iterative optimization.
- Clarification targets: Derived from discussions between frontline practitioners and FDEs.
- Case-specific evaluators: Assess final artifacts against acceptance criteria using FDE solutions as references.
- Delivery score: A metric measuring the quality of the final artifact compared to the FDE reference.
- Delivery success rate (DSR): The fraction of planned runs reaching the FDE reference.
FAQ
What is FDE-Bench?
FDE-Bench is a benchmark designed to evaluate agents on end-to-end delivery tasks that require interactive requirement clarification and iterative optimization in real-world business scenarios.
What are the key challenges addressed by FDE-Bench?
FDE-Bench addresses the challenge of handling real-world business requests that are often underspecified, requiring agents to elicit missing requirements and iteratively optimize solutions to meet customer acceptance criteria.
What are the results of the FDE-Bench evaluation?
The highest mean delivery score achieved by agents is 0.5510, compared to an FDE reference of 1.0, and the highest delivery success rate is 6.12%, indicating significant room for improvement in current AI agents.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.AI on Oct 14, 2026.
Comments
Sign in to join the discussion.