Speculative decoding

2 stories

Stories

With software alone, one B200 beats the LPU and gets close to Cerebras
Stock SGLang served gpt-oss-120b at 411 tokens per second on one rented B200. Four settings took the same card and the same weights to 1,366, and single prompts to 2,215. No kernel written, nothing recompiled.
RunInfra BlogAug 6, 202617 min read
Lossless Inference
How to make LLM serving faster without touching the model. Exact kernels, speculative decoding, lossless compression, KV reuse and scheduling, with the math and how to verify it with logit parity.
RunInfra BlogAug 3, 20268 min read

You're all caught up.

AiByts Weekly Brief

Get the top AI news and tools straight to your inbox every week.

Weekly · AI news

Join 15,000+ AI enthusiasts