ASSET: Adaptive Speech Segmentation via Reinforcement Learning
Speech large language models (LLMs) are often built using pretrained acoustic encoders as an audio interface to an LLM decoder. However, LLMs scale poorly with input sequence length, while acoustic encoders tend to produce substantially longer sequences than typical textual inputs.
This paper introduces ASSET, a reinforcement learning framework designed to improve the efficiency of speech large language models (LLMs) by adaptively compressing acoustic encoder outputs.
What happened
The article presents ASSET, a reinforcement learning-based method for adaptive speech segmentation. It addresses the challenge of mismatched sequence lengths between acoustic encoders and LLM decoders by compressing audio data while preserving task-relevant information.
Why it matters
Efficient speech processing is critical for real-time applications. ASSET's compression technique reduces computational overhead without sacrificing performance, making it valuable for optimizing speech LLMs in practical deployments.
- Reinforcement learning can effectively optimize audio data processing pipelines
- Adaptive segmentation improves efficiency without compromising task performance
- The framework's flexibility allows adaptation to diverse speech-related tasks
The bigger picture
The method provides a scalable solution for integrating speech inputs with LLMs, potentially improving efficiency in voice assistants, transcription systems, and other audio-based AI applications.
ASSET offers a novel approach to speech-LLM integration by addressing sequence length mismatches through RL-driven compression, demonstrating practical benefits for both ASR and non-ASR applications.
Technical details
- Reinforcement learning (RL) framework for adaptive segmentation
- Group-relative policy optimization (GRPO) for training
- Variable-duration segment partitioning
- Continuous audio token pooling mechanism
- Task-performance feedback loops with token-rate penalties
FAQ
What problem does ASSET solve?
ASSET addresses the challenge of mismatched sequence lengths between acoustic encoders and LLM decoders by compressing audio data while preserving task-relevant information.
How does ASSET differ from traditional speech processing methods?
Unlike fixed-length encoding approaches, ASSET uses reinforcement learning to dynamically segment and compress audio sequences, adapting to task requirements.
What are the key performance metrics demonstrated?
ASSET achieves ASR performance comparable to transcript-assisted oracles while reducing audio token counts by ~25% on English LibriSpeech and improving on phone-boundary oracles for Mandarin AISHELL.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.CL (NLP) on Oct 14, 2026.
Comments
Sign in to join the discussion.