Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

The paper challenges the assumption that language models need explicit tokenizers to be efficient demonstrating that standard flat Transformers can process raw byte sequences and actually outperform traditional subword models as parameter sizes scale.