An AI token is a small piece of text that a language model can process. A token may contain a whole word, part of a word, a space, or a punctuation mark. Models do not read a sentence as a human sees it. A tokenizer first breaks the sentence into these pieces and replaces each piece with a number.
A token is not the same as a word
Short common words often fit into one token. A longer or unusual word may split into several. The word “unhelpfulness” might become pieces that resemble “un,” “help,” “ful,” and “ness.” Exact splits depend on the tokenizer. Two models can divide the same line in different ways.
Spaces matter too. Many tokenizers keep a leading space with the word after it. Emojis, source code, and text written in some languages may use more tokens than an English sentence with the same visible length. A rough words-to-tokens rule can help with planning, but only the model’s own tokenizer gives an exact count.
Why models use pieces
A model needs a fixed vocabulary of items it can recognize. Storing every possible word would fail as people invent names, combine terms, and switch languages. Small text pieces let a limited vocabulary represent a huge range of writing. The model can also notice useful parts shared by related words.
During generation, the model predicts a likely next token. It does not pick a finished paragraph in one step. Each new choice joins the input, then the process repeats. A visible word can arrive through one prediction or several predictions.
Tokens set practical boundaries
Language model services often count both input and output tokens. The input includes the user’s request and any instructions or documents sent with it. The output includes the model’s reply. Some services set prices by token count. A long report can cost more because the model has more pieces to process and produce.
The context window also uses tokens as its unit. A model with a 100,000-token window can consider up to that amount at once. The total may include hidden system instructions, chat history, uploaded text, and the answer in progress. Reaching the limit can force an application to remove old material or shorten documents.
Token counts can surprise developers
A page that looks short may contain markup, repeated data, or code that creates many tokens. Sending an entire database record for a simple question wastes space. Good applications select the relevant text and keep instructions direct. This can reduce delay and cost without changing the model.
Tokens are units for computation. They do not show how hard an idea is. Ten tokens can contain a difficult math request. A thousand tokens can hold a simple list. Counting helps manage capacity, but it does not measure truth, quality, or understanding.