Inference
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The phase where a trained model is put to work: it applies its learned parameters to new data and returns predictions, one request at a time.
What is Inference?
Inference is the phase in which a trained machine learning model is put to work: it takes new, unseen data and applies its learned parameters to produce a prediction, classification, or recommendation.
In statistics, “inference” means drawing conclusions about a population. In machine learning it means something narrower: running a finished model on live input. When engineers talk about inference cost or inference latency, this serving phase is what they mean.
How Inference Works
At inference time the model’s parameters are frozen. Input arrives, flows forward through the model’s computations, and comes out as an output: a label, a score, a sentence. No learning happens; the model gives the same answer to the same input every time.
Because inference runs on every single request, its cost compounds. A model queried a million times a day pays its inference bill a million times, which is why teams compress models with quantization or prune them for sparsity before serving them at scale or on edge devices.
Inference vs Model Training
Inference is when the model answers; model training is when it learns. The two phases have opposite economics: training is a large periodic compute bill, while inference is a small cost paid on every request for the life of the system.
| Inference | Model training | |
|---|---|---|
| What happens | Parameters are frozen; the model only produces outputs | Parameters change with every batch of examples |
| When it runs | Every time the deployed model receives a request | Before deployment, and again for retraining |
| Compute profile | Single requests, where milliseconds of latency matter | Large batches on GPU clusters, for hours or weeks |
| Typical failure | Latency spikes, data drift, serving a stale model | Overfitting, data leakage, mislabeled data |
Example of Inference
A bank runs a fraud model trained months ago on historical transactions. You tap your card at a store, and the transaction details – amount, merchant, location, time – become the model’s input.
The model applies its frozen parameters and returns a fraud score in a few milliseconds. Above a threshold, the transaction is blocked; below it, the payment clears before the terminal finishes beeping.
That single scoring pass is inference. The model learned nothing from your transaction; it will only get smarter when the bank retrains it on fresh data.
Related AI terms: Model Training · Quantization · Deploy · Edge Computing
Did you like the Inference gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About