I animated what GPT-4o's tokenizer does to "strawberry": real token ids, a tiny BPE run, and why the space belongs to the word (6 min video) [D]
I made this explainer for my small channel and would like feedback from people learning this, especially if anything is wrong.
Reddit - r/ML1 min read