Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization.