Know a great AI tool or AI news source? Submit it to AiByts
SGLang10 min read

Why Your Self-Hosted vLLM Server Never Says No

vLLM 0.33 ships a request cap, — max-num-queued-reqs, that is off by default. In a simulated 40-call agent burst on 4 slots, a tight cap refused 33 calls and cut P95 time-to-first-token for the rest from 37 s to 2.3 s.

By Chew Loong Nian - AI ENGINEERTowards AI

Read the full story

Originally published by Towards AI on Nov 10, 2026.

Read on Towards AI(opens in a new tab)

Comments

Sign in to join the discussion.

No comments yet — be the first to share your thoughts.