vLLM2 min read
Cancellation Is Not Enough: What Retries Leave Behind in an Overloaded LLM Engine
Clients and proxies in front of an LLM engine retry calls that time out. After a transient overload, these retries can keep the engine from recovering even when calls arrive below its capacity.
By Xindan Zhang, Fei WangarXiv - cs.AI
Read the full story
Originally published by arXiv - cs.AI on Nov 9, 2026.
Comments
Sign in to join the discussion.