Scale, Drivers, and Limits of Representational Alignment Across Large Language Models
Comparing large language models can reveal shared representational structure and systematic differences between them. However, aggregate similarity measures can obscure how agreement varies across geometric scales and which linguistic properties support it.
This paper introduces methods to analyze representational alignment between large language models, examining how models agree on linguistic structures at different geometric scales and what linguistic properties drive this alignment.
What happened
Researchers developed Neighborhood-Weighted RSA (NW-RSA) to measure alignment across different representational scales and Counterfactual RSA (CF-RSA) to test linguistic property impacts. Experiments across multiple language models revealed varying degrees of agreement depending on input proximity and linguistic features.
Why it matters
Understanding representational alignment helps improve model consistency, interpretability, and cross-model collaboration. The findings inform how linguistic properties influence model behavior and highlight limitations in assuming universal representational structures.
- Model alignment varies significantly between local and global representational scales
- Linguistic perturbations reveal selective sensitivity in model alignment
- Natural language structure plays a critical role in representational convergence
- Standard similarity measures may obscure important alignment differences
The bigger picture
The methods provide new tools for analyzing model alignment, influencing future research on model consistency and cross-architecture comparisons. The findings challenge assumptions about universal representational structures in LLMs.
The study advances understanding of how large language models represent linguistic structures, demonstrating that while broad geometric alignment exists, finer-grained differences persist and depend on specific linguistic properties.
Technical details
- Neighborhood-Weighted RSA (NW-RSA): Weights stimulus pairs by their proximity in representational space
- Counterfactual RSA (CF-RSA): Compares intact representations with perturbed versions to test linguistic sensitivity
- Representational similarity analysis (RSA) framework for comparing model representations
FAQ
What are NW-RSA and CF-RSA?
NW-RSA (Neighborhood-Weighted RSA) measures alignment across different representational scales by weighting stimulus pairs based on proximity. CF-RSA (Counterfactual RSA) tests linguistic property impacts through controlled perturbations.
What did the study find about model alignment?
Models show strong agreement in coarse geometric structures but significant divergence in local neighborhoods. Alignment is more sensitive to changes in logical scope than syntactic or structural modifications.
Why is this research important for AI development?
It provides tools to analyze model consistency and reveals how linguistic properties influence representational alignment, informing improvements in model design and interpretation.
This summary and analysis were generated with AI from the original article and may contain errors. Read the original for the full story.
Read the full story
Originally published by arXiv - cs.CL (NLP) on Oct 14, 2026.
Comments
Sign in to join the discussion.