Cooper: My week is mostly tabs arguing with other tabs, which feels appropriate, because the thing I kept seeing in every tab was tokens. Token budget, token routing, token cost, tokenized this, tokenized that. And we have been very guilty of saying tokenization like it's obvious.
Miles: Yeah, we do the classic Exploring Next move where we say the load-bearing word quickly and then sprint away from it. We keep using it because it sits underneath almost every model conversation. Context windows, cost, retrieval, agents, even weird image generation stuff.
Cooper: Exactly. And enough people have poked us with the very fair version of, okay, can you stop doing the expert nod and actually explain the thing? So this one gets the slow treatment, Miles. Tiny scissors out.
Miles: Tiny scissors is actually not bad. The mental model I want is: imagine a sentence as a strip of material, and the tokenizer is the little cutting machine that chops it into labeled tiles. Each tile comes from a catalog the model knows. The model doesn't read the ink the way we do. It sees the catalog numbers stamped on the tiles.
Cooper: So in that tile picture, if I type, say, the sentence, the model isn't getting my sentence as a sentence. It's getting tile number seventeen, tile number eight thousand something, tile number whatever, in order.
Miles: Yes. And before the real terms show up, there are three small pieces to make plain. Text representation just means choosing a form for text that a computer can store and manipulate. Discrete symbols means separated units you can count, like tile one, tile two, tile three. Neural network input means the actual numbers fed into the model, because a neural network is basically a big mathematical function that operates on numbers.
Miles: Tokenization is the bridge across those three. It takes raw text, turns it into discrete symbols called tokens, then maps those tokens to integer identifiers. Those identifiers are what the rest of the model pipeline can work with.
Cooper: I love that the bridge to artificial intelligence is, fundamentally, a very fussy label maker.
Miles: A fussy label maker with consequences. The tokenizer has a vocabulary, which is the catalog of all tiles it knows how to use. The vocabulary says this string maps to this I D, that string maps to that I D, and so on. Once the model is trained, that mapping is not something you casually swap out.
Cooper: Because then the numbers mean different tiles.
Miles: Exactly. If the model learned that I D four thousand and twelve usually behaves like one text piece, and you hand it a different tokenizer where four thousand and twelve means something else, you've scrambled the inputs. It is like relabeling the whole parts bin after the factory workers trained on the old labels.
Cooper: That part is very product-real. A tokenizer is not some preprocessing footnote you can update like a color theme. If a team changes it under a model, the model can get worse for reasons that look spooky until you notice the tile catalog changed.
Miles: And this is where people often imagine tokens as words, which is understandable but usually wrong for modern large language models. A large language model is the kind of model trained to predict and generate text, like the GPT family, BERT-style models, or Llama models. In those systems, a token might be a whole word, but it might also be a chunk of a word, a punctuation mark, a space plus a word fragment, or some other learned piece.
Cooper: Wait, slow that down. Why not just make each word a tile? That sounds simpler.
Miles: It is simpler until language does language things. Whole-word tokenization needs a vocabulary entry for every word you expect to see. Then it runs into rare words, new product names, typos, code, usernames, weird capitalization, and words in languages the tokenizer saw less often.
Miles: Character-level tokenization is the opposite extreme. Every character is a tile, so it can represent almost anything. But now a short sentence becomes a lot of tiles, and the model has to process a much longer sequence. That costs more memory and compute.
Cooper: So subword tokenization is the compromise. Not every word gets its own tile, and not every letter is its own tile. You get reusable chunks, like prefixes, suffixes, common word pieces, common punctuation patterns.
Miles: Yes. Modern tokenizers usually live in that middle zone. The common vocabulary size is often somewhere around fifty thousand to one hundred thousand tokens, which is big enough to include frequent pieces but not so big that the catalog becomes absurd.
Cooper: And that is the point where Byte Pair Encoding enters, right? B P E, which we have absolutely name-dropped like goblins.
Miles: Cooper, yes. Byte Pair Encoding is one way to learn the tile catalog. The plain version is: start with very small pieces, then repeatedly merge pairs of pieces that show up together a lot in the training text. If two character sequences keep appearing next to each other, the tokenizer can treat that pair as one bigger token.
Cooper: Okay, so it learns that some cuts are annoying to keep making. If the same two tiles are always snapped together, just manufacture them as one tile.
Miles: That's the tile factory version, yes. There are related approaches with names like WordPiece and SentencePiece. The shared idea is learning useful subword units so the model can handle new text while keeping the sequence from exploding.
Cooper: I want to sit on sequence length for a second, because this is where tokenization stops being vocabulary trivia and starts becoming money. Last episode, when we did context windows, we said a context window is the amount of text the model can actively process at once. The hidden gotcha was that the window is counted in tokens, not in vibes, not in pages, not even cleanly in words.
Miles: Yeah.
Cooper: So if two people paste the same apparent amount of text, but one version tokenizes into way more pieces, one of them is burning more of the model's working room.
Miles: That is the practical bite. The transformer part of many language models compares tokens to other tokens to figure out what matters for the next prediction. More tokens means a longer sequence to process, and that can get EXPENSIVE in memory and compute.
Cooper: And it explains why a sentence can feel short to a human but cost more than expected. Code is a classic one. Weird punctuation, lots of symbols, long identifiers, file paths you should not read aloud on a podcast because we're not monsters.
Miles: Spaces matter. Some tokenizers represent leading spaces as part of a token. Some handle whitespace and punctuation differently. That is why two tokenizers can split the same sentence in noticeably different ways.
Cooper: Back to seriousness, somehow. This is also why English-centric training data can create uneven cost across languages, right?
Miles: Yes. If a tokenizer learned its vocabulary mostly from English-heavy text, it may have nice big reusable tiles for common English pieces and much smaller, choppier tiles for other languages. The model can still process the text, but it may take more tokens to represent the same meaning.
Cooper: That one always bothers me because it is invisible from the interface. A user just sees that the tool is slower, pricier, or weirdly worse in one language, and the cause may be down in the cutting machine.
Miles: Invisible is the key word. Tokenization feels like plumbing until it changes behavior. Punctuation, capitalization, whitespace, and rare words can all change the token sequence. The model learns patterns over those sequences, so those little cuts can matter.
Cooper: This is where I want to pull in some concrete things we have looked at. The agent tool-routing piece, the one with the huge token-use reduction claim, was basically about not dumping every possible tool description into the model every time. Tool descriptions are not free prose floating in the air. They become tokens, and tokens are the bill and the latency.
Miles: Right. I was skeptical of the headline number there, because I am spiritually obligated to side-eye ninety-nine percent claims. But the mechanism made sense: retrieve or route to a smaller relevant set of tool descriptions so the model receives fewer tokens. The model did not magically become smarter. The system sent fewer tiles down the belt.
Cooper: Another one is the HTML versus Markdown thing we talked through. The argument was that HTML can preserve structure better in agentic loops, especially when humans need to inspect or guide what the agent is doing. Tokenization sits underneath that too, because markup has its own symbols and nesting and repeated patterns.
Miles: Yes, though I would be careful not to say HTML is always cheaper or always better. The representation changes the token stream. Sometimes extra structure helps the model and the human stay aligned. Sometimes it bloats the sequence. Tokenization is where that trade-off becomes measurable rather than aesthetic.
Cooper: And then there is the image example, which sounds like cheating because we said text. That masked discrete diffusion image model we looked at was using discrete image tokens, not word tokens. Different domain, same broad move: turn messy continuous stuff into countable pieces the model can manipulate.
Miles: Exactly. For images, the tokens are not subwords. But the prerequisite idea is the same: the model is working over discrete symbols. That is why tokenization is broader than text in machine learning, even though text tokenization is the one people usually mean when they complain about context windows.
Miles: That lands. I would only add one caution: the cut is learned or designed before the model runs, and it becomes part of the system's behavior. Not glamorous. Very load-bearing.
Cooper: Miles, the fussy label maker got its whole episode, and somehow you made me respect it. Exploring Next remains deeply normal.