PBT doesn’t make grokking faster, but it can escape a hyperparameter range where nothing groks [P]
I ran Population Based Training over learning rate and weight decay on grokking (modular addition mod 97, 35% train split, small MLP, 20 members, 20k steps, full batch).
Reddit - r/ML2 min read