SpecRelay: Amortize or Avoid the Round Trip in Edge-Cloud Speculative Decoding
Edge-cloud speculative decoding moves autoregressive drafting to an edge device while a larger cloud model verifies draft tokens in parallel, so that edge drafting and cloud verification overlap under a synchronous draft-verify schedule.
arXiv - cs.AI2 min read