Models Trained to Evade Topic Probes Struggle to Answer Questions about Those Topics

TL;DRNeural Chameleons are models trained to evade activation probes whenever the prompt says "You are being probed for X". The paper reports that this generalizes to probes the model never saw, at a small capability cost (about 3 points on MMLU and 1 on GSM8K).