A transformer is a type of neural network that finds relationships between parts of an input. It became the main design behind many large language models. Transformers also work with images, audio, video, computer code, and biological sequences.
The design relies on attention. Attention lets the model give more weight to pieces that help with the current calculation. In the sentence “Maya put the glass on the table because it was level,” the word “it” likely connects to the table. Attention helps the model build that connection.
Tokens enter together
A tokenizer splits text into pieces and maps each piece to numbers. The model adds position information so it can tell which piece came first. Without that signal, “dog bites person” and “person bites dog” would contain the same pieces with no clear order.
Earlier sequence models often processed text one step after another. Transformers can examine many positions in parallel during training. That made it practical to learn from far larger datasets on modern hardware.
Attention builds a new view of each piece
For every token, the model creates three kinds of vectors called a query, key, and value. The query asks what information this position needs. Keys describe what other positions can offer. A comparison between queries and keys sets attention scores. The model mixes the values according to those scores.
Multi-head attention repeats the process through several learned views. One head may track nearby grammar. Another may connect a name to a pronoun. These roles come from training rather than labels written by a developer.
A transformer stacks attention layers with other neural network layers. Early layers may capture local patterns. Deeper layers can form richer links. The final output depends on every calculation along that path.
Encoder and decoder designs
An encoder reads input and creates a representation of it. Encoder models suit tasks such as classification and search. A decoder predicts the next token from earlier tokens. Many text generators use decoder-only models. An encoder-decoder system reads one sequence and creates another. Translation and summarization often use that shape.
A decoder uses a mask during training so it cannot look at future words while predicting the next one. It learns the same left-to-right task used during generation. The model produces one token, adds it to the sequence, and repeats.
Power comes with a cost
Standard attention compares many token pairs. Memory and computing needs rise fast as the input grows. Researchers use optimized attention, smaller precision numbers, caching, and other designs to manage long context. The model still needs substantial hardware at a large scale.
A transformer does not check facts by itself. It learns statistical relationships from data. Bias, missing examples, and misleading patterns can shape its output. The architecture explains how information moves through the model. It does not guarantee that the result is correct.