Evaluating LLMs as Agentic Workflow Executors in Real-World Customer Support Skills
Enterprise customer-support agents increasingly rely on Large Language Models (LLMs) to execute bounded, tool-using workflows. We study this problem through the lens of Skill LLMs: specialized executors that operate within a selected skill context and must complete a narrow, stateful workflow under…
This study evaluates how well large language models (LLMs) execute structured workflows in enterprise customer support scenarios, focusing on their ability to follow tool-based instructions within constrained contexts.
What happened
Researchers developed a benchmark to assess LLMs' ability to execute stateful, tool-using workflows in customer support contexts. They evaluated 17 models using multiple judges and a rule-based evaluator, revealing performance rankings and highlighting the importance of evaluation methodology.
Why it matters
For AI developers and enterprises deploying LLMs in customer support, this study provides insights into model capabilities, evaluation reliability, and the trade-offs between closed and open-weight models in real-world workflows.
- Benchmarking is critical for assessing LLMs in real-world workflow execution
- Judge reliability significantly affects evaluation outcomes in zero-shot testing
- Open-weight models can achieve competitive performance against closed models
- Workflow execution requires careful handling of tool constraints and state management
The bigger picture
The findings inform model selection for customer support applications and emphasize the need for standardized evaluation methodologies that account for human evaluator variability.
The study establishes a framework for evaluating LLMs in customer support workflows, revealing performance hierarchies and methodological considerations that impact assessment reliability.
Technical details
- Model Context Protocol (MCP)-compatible mocks for environment state simulation
- Structured skill descriptions with policy and tool constraints
- Zero-shot evaluation framework with tool coverage, hallucination, and claim validation checks
- Kappa statistic for measuring judge agreement in evaluation processes
FAQ
What is a 'Skill LLM' according to the study?
A specialized LLM that operates within a specific skill context to complete stateful, tool-using workflows under policy constraints.
How were models evaluated in this study?
Through a benchmark pipeline that included tool coverage checks, hallucination detection, and multi-turn dialogue simulations, with evaluations by human judges and a rule-based system.
What does the Kappa score indicate in this context?
The level of agreement among human judges in pass/fail evaluations, with 0.524 average agreement across 10 judge pairs.
Which model performed best in the evaluation?
GPT-5.4 led the rankings, while QWEN3.5-9B was the strongest open-weight model tested.
What technical challenge does the study address?
The reliability of zero-shot evaluation methods for LLMs executing complex, tool-dependent workflows in enterprise settings.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.CL (NLP) on Oct 14, 2026.
Comments
Sign in to join the discussion.