Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization
On-Policy Self-distillation is a self-improvement paradigm that enables large language models (LLMs) to improve their capabilities by leveraging the same model as both teacher and student under different conditioning contexts.
This paper addresses capability degradation in large language model (LLM) self-improvement through a novel technique called Representation Geometry Regularization (RGR).
What happened
The article identifies a problem where LLMs trained via OPSD lose previously acquired capabilities during optimization. The authors propose RGR to stabilize hidden representation geometry, demonstrating its effectiveness across multiple benchmarks without altering existing OPSD workflows.
Why it matters
For AI developers and researchers, this addresses a critical challenge in maintaining model reliability during self-improvement processes, ensuring sustained performance across diverse tasks.
- Capability degradation occurs in OPSD despite overall performance gains
- Maintaining representation geometry is critical for stable model improvement
- RGR offers a non-invasive way to enhance existing self-improvement frameworks
The bigger picture
The method provides a practical solution to stabilize LLM training processes, potentially improving the reliability of self-improving AI systems in real-world applications.
The paper presents RGR as an effective approach to mitigate capability degradation in LLM self-improvement by preserving the geometric structure of hidden representations.
Technical details
- On-Policy Self-distillation (OPSD): A self-improvement method using the same model as both teacher and student
- Representation Geometry Regularization (RGR): Technique to preserve hidden representation structure
- Intra-cluster compactness: Measures how tightly related data points are grouped
- Inter-cluster separation: Measures distinctness between different data clusters
FAQ
What is On-Policy Self-distillation (OPSD)?
OPSD is a self-improvement technique where LLMs use their own outputs as both training targets (teacher) and learning inputs (student) under varying conditions.
Why does capability degradation occur in LLMs?
Capability degradation happens when previously correct responses become incorrect during training, linked to deteriorating hidden representation geometry.
How does RGR work?
RGR maintains the geometric structure of hidden representations by enforcing compactness within clusters and separation between clusters during post-training.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.AI on Oct 12, 2026.
Comments
Sign in to join the discussion.