NLP Transformer
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A neural network architecture that uses self-attention to process all words in a sequence at once, the foundation of GPT, BERT, and modern language models.
What is an NLP Transformer?
A transformer is a deep learning architecture that uses self-attention to process every word in a sequence at once, weighing how much each word matters to every other.
Introduced in “Attention Is All You Need” (Vaswani et al., 2017), it replaced the step-by-step reading of recurrent networks with parallel processing. That change made training on internet-scale text practical, and every major language model since – GPT, BERT, and their successors – is a transformer.
How a Transformer Works
- Self-attention: For each word, the attention mechanism scores its relevance to every other word in the input. In “the animal didn’t cross the street because it was tired”, attention links “it” to “animal” regardless of the distance between them.
- Multi-head attention: Several attention computations run in parallel, each free to track a different kind of relationship, such as syntax in one head and coreference in another.
- Positional encoding: Because all words are processed simultaneously, the model adds a position signal to each word so that order is not lost.
- Encoder and decoder stacks: The original design pairs an encoder that reads the input with a decoder that generates output. Many descendants keep only one half: BERT is encoder-only, GPT is decoder-only.
Applications of Transformers
- Text generation: GPT-style models write answers, code, and documents one predicted token at a time; how that works in practice is covered in how ChatGPT really works.
- Machine translation: The task the architecture was designed for, as a sequence-to-sequence system.
- Search and question answering: Encoder models like BERT score how well a passage answers a query.
Example of an NLP Transformer
Trace one exchange with a transformer chatbot. The user writes: “I ordered a blue lamp last week. It arrived broken. What should I do?”
Tokenized input flows through the attention layers, where “It” attaches to “lamp” and “broken” attaches to “arrived”. The model now generates a reply token by token, each new token attending back over the whole conversation, which is why the answer addresses a broken lamp and not some generic complaint. Earlier LSTM-based systems compressed the request into a single fixed vector and routinely lost details like the color or the timeframe.
Related AI terms: Attention Mechanism · Sequence-to-Sequence · LSTM · GPT · BERT
Did you like the NLP Transformer gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About