How ChatGPT Works: From Next-Token Prediction to Agents
Last Updated: September 10, 2026 | By Mihail Sebastian | ChatGPT vs Friends
ChatGPT wraps a large language model in post-training, tools, memory, and agent modes. Here is what happens between your message and the reply.
Ask ChatGPT to explain itself and you get a fluent answer. The honest version starts with a distinction most explainers skip: ChatGPT is a product wrapped around large language models, and in 2026 the wrapping, the tools, memory, and agent machinery bolted onto the model, matters as much as the model itself.
This article walks through both halves. First the base skill every layer depends on, then the layers, then what actually happens in the seconds between your message and the reply.
The Base Skill: Predicting the Next Token
Underneath everything sits one trick: predicting the next token, roughly a short word or a piece of a longer one. During pretraining, the core phase of model training, the model reads an enormous body of text and adjusts its internal parameters, millions of small corrections at a time, to get better at that single prediction.
The architecture that made this work at scale is the transformer. Its attention mechanism weighs every token in the input against every other, so “it” in a long sentence attaches to the right noun, and a question at the end of a conversation connects back to a detail from the start.
Nobody programs in grammar, facts, or coding style. They emerge because knowing them makes the next-token prediction more accurate.
The same origin explains the core weakness. The model is trained to produce plausible text, not verified text, and a fluent hallucination is exactly what a plausibility machine produces when the fact is missing. Fluency and truth are separate properties, and only one of them was the training objective.
From Raw Predictor to Assistant
A freshly pretrained model is not a chatbot. Ask it a question and it might continue with three more questions, because that is a statistically likely continuation of question-shaped text.
Post-training closes the gap. Instruction tuning trains the model on examples of requests paired with good responses, so it learns the shape of being helpful. Then human feedback sharpens behavior: people compare candidate answers, and a reinforcement process steers the model toward the answers people prefer and away from the ones they reject.
One more layer sits between you and the model. Every conversation begins with system rules from the developer, instructions about tone, safety, and tool use that you never see and that outrank your requests. When ChatGPT declines to do something you asked for, you have hit that layer, not a limit of the model’s ability.
The Layers 2024 Explainers Miss
Most explanations of ChatGPT stop at the model. The product has kept growing, and four additions changed what “how it works” means.
Reasoning modes. For hard problems, the model spends extra compute thinking before it answers: it drafts intermediate steps, checks them, revises, and only then writes the reply. You trade time and cost for fewer mistakes on math, code, and multi-step logic, and ChatGPT decides, or lets you decide, when the trade is worth it.
Tools. Through tool calling, the model requests actions it cannot perform itself: searching the web, executing code, reading a file you uploaded. The model never runs anything directly. It emits a structured request, the platform executes it, and the result flows back into the conversation for the model to use.
Memory. ChatGPT carries information across conversations: your preferences, ongoing projects, standing instructions. The old claim that it cannot learn from interactions is out of date at the product level. The model’s weights still do not change as you chat, but the product remembers, and you control it: you can review what is stored, edit it, or switch it off.
Agent modes. Given a goal, ChatGPT works through multi-step tasks on its own: browsing, filling forms, chaining tool calls, and checking back when it needs a decision. Mechanically this is tool calling in a loop, with the model choosing the next step after reading each result.
What Happens When You Hit Enter
Trace one message through the stack. You upload a contract, type “Summarize this and flag any auto-renewal clause,” and hit enter.
- Tokenize. Your text is split into tokens, the units the model reads and writes.
- Assemble the context. The system rules, relevant memory, the conversation so far, and your file are packed into one input sequence.
- Forward pass. The sequence flows through the model’s frozen parameters, this is inference, and out comes a probability distribution over every possible next token.
- Sample. One token is picked from that distribution, appended to the sequence, and the pass runs again. Generation is this loop, thousands of times per reply.
- Call a tool if needed. If the model decides the contract needs a search or a calculation, it emits a tool call mid-generation, waits for the result, and continues with it in context.
- Stream. Tokens are sent to your screen as they are produced, which is why the answer appears to type itself.
The sampling step explains a familiar oddity: ask the same question twice and you get two different answers. The model produces a distribution, not a single verdict, and a controlled amount of randomness in the pick keeps replies varied instead of repetitive. Nothing changed its mind between your two attempts; the dice simply landed differently.
Every capability in the previous section is a variation on this loop. Reasoning modes run it longer before showing you anything; agent modes run it repeatedly with tool results in between. Memory changes what gets packed into step two, and system rules sit at the front of the sequence on every single pass.
The Limits That Persist
The layers changed the product; they did not change the physics. Hallucination persists because prediction is still the engine: a confident, specific, wrong answer remains a natural output, and the invented auto-renewal clause looks exactly like a real one.
The knowledge gap persists too. The model’s built-in knowledge ends at its training cutoff, and live search only closes the gap when it actually runs and when the sources it fetches are good. An answer delivered without a search behind it is an answer from a snapshot of the past.
Above all, tone tells you nothing. The model writes with the same assurance whether it is reciting settled fact or fabricating a citation, so confidence is a style, not a signal. Courts have already sanctioned lawyers who filed briefs full of case law ChatGPT invented, and every one of those citations read as authoritative.
The working posture that follows is simple. Treat ChatGPT as a fast, capable drafter whose work you check: ask it to search and cite sources when facts matter, verify the citations exist, and let it be wrong only where wrong is cheap. Understanding the loop underneath, prediction dressed as conversation, is what makes that discipline feel less like paranoia and more like knowing how the machine works.
