ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration
Serving an LLM efficiently depends on the engine, how the model is split across GPUs, precision, scheduler settings, speculative decoding, KV-cache management, and prefill-decode disaggregation. The best configuration depends on the model, hardware, workload, and objective.
arXiv - cs.AI2 min read