Tokenization
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
Splitting text into tokens (words, subwords, or characters) so a language model can turn it into numbers – the first step of every NLP pipeline.
What is Tokenization?
Tokenization is the process of splitting text into smaller units, called tokens, that a language model converts into numbers and processes. Tokens are whole words, pieces of words, or single characters.
It is the first step of nearly every natural language processing pipeline. A model never sees letters or sentences, only token IDs – which is why AI providers bill by the token, and why models are surprisingly bad at counting the letters in a word.
Types of Tokenization
- Word tokenization splits text into whole words: “I love AI” becomes [“I”, “love”, “AI”]. It is simple, but the vocabulary grows enormous and any unseen word becomes an unknown token.
- Subword tokenization breaks rare words into reusable pieces: “unhappiness” becomes [“un”, “happiness”]. Modern language models, including GPT, use subword schemes such as byte pair encoding, which cover any text with a vocabulary of modest size.
- Character tokenization splits text into individual characters: “AI” becomes [“A”, “I”]. The vocabulary is tiny and nothing is ever unknown, but sequences get long and each token carries little meaning.
Example of Tokenization
A sentiment model receives the review “Unbelievable turnaround!”. Its subword tokenizer has no single entry for “Unbelievable”, so it splits the text into pieces it knows: [“Un”, “believ”, “able”, " turnaround", “!”].
Each piece maps to an ID in the tokenizer’s vocabulary, say [512, 8067, 411, 15230, 0]. Those IDs go to the embedding layer, which turns each one into a vector the network computes on. From this point forward the model works with five numbers, not two words.
The same text produces different tokens under different tokenizers. That is why token counts, context limits, and API costs vary from one model to the next even for identical input.
Related AI terms: Natural Language Processing · Word Embedding · Embedding Layer · NLP Transformer
Did you like the Tokenization gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About