PreSO: Privileged Representation Isolation for On-Policy Self-Distillation
On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learning with verifiable rewards (RLVR) into dense token-level supervision.
This paper introduces PreSO, a method for improving on-policy self-distillation in reinforcement learning by isolating privileged information through representation-based techniques.
What happened
The article presents PreSO, a technique that modifies on-policy self-distillation by isolating privileged information through representation-based constraints. It compares solution-specific and hint-induced representation shifts, demonstrating that hint-based approaches yield better performance despite revealing less information.
Why it matters
The method provides a framework for more efficient knowledge transfer in reinforcement learning systems, particularly for tasks requiring dense supervision from sparse rewards. Its effectiveness across multiple model scales suggests broad applicability for AI development teams.
- Privileged information can be effectively isolated through representation constraints
- Compact hints produce more transferable knowledge than full solution details
- Low-rank subspace estimation enables efficient knowledge distillation
- The method shows consistent benefits across different model sizes
The bigger picture
The approach demonstrates measurable improvements in model accuracy and training consistency, offering a practical enhancement for reinforcement learning systems in structured problem domains like mathematical reasoning.
PreSO offers a novel approach to on-policy self-distillation by focusing on reusable representation patterns, achieving state-of-the-art results in mathematical reasoning tasks through improved knowledge transfer mechanisms.
Technical details
- Privileged Representation Isolation (PreSO)
- On-policy self-distillation (OPSD) with token-level supervision
- Low-rank subspace estimation from cross-problem representation shifts
- Compact hints for limiting solution-specific information in privileged shifts
FAQ
What problem does PreSO solve?
PreSO addresses the challenge of distilling privileged information from teachers to students in reinforcement learning, where the teacher's advantage may rely on information unavailable to the student during inference.
How does PreSO differ from traditional self-distillation methods?
PreSO uses compact hints to limit solution-specific details in privileged shifts and employs low-rank subspace estimation across problems to create more transferable representation-level distillation targets.
What are the key performance results?
PreSO improves average accuracy by 6.8-10.0 points over base models across Qwen3-1.7B, 4B, and 8B, with sustained improvements throughout training.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv cs.LG on Oct 14, 2026.
Comments
Sign in to join the discussion.