Embeddings Are Overpriced? A Frozen 16 MB Vocabulary for Small and Ternary Language Models

In a small language model the vocabulary is the single largest part. In Gemma 3 270M the embedding table holds 168M of 268M parameters, and because quantisation methods leave the embedding in 16-bit, it is also most of the file once the rest of the model is compressed.

Analyze with AIAnalyze with ChatGPTAnalyze with ClaudeAnalyze with PerplexityAnalyze with Google AI ModeAnalyze with GrokAnalyze with Microsoft Copilot