How can LLM RL Work Despite Information-Theoretic Inefficiency

Cross-posted from my personal blogThe success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of…