Optimizer
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The training algorithm that updates a neural network's weights step by step during learning, steering them toward values that minimize the loss function.
What is an Optimizer?
An optimizer is the algorithm that updates a model’s weights during training. After each batch of data, the loss function reports how wrong the model currently is; the optimizer decides how to change every weight in response.
Training a network is nothing more than running this loop millions of times, so the choice of optimizer shapes how fast training converges and whether it converges at all.
Types of Optimizers
- Stochastic Gradient Descent (SGD): The baseline. Stochastic gradient descent moves each weight a small step against its gradient, one mini-batch at a time. Simple, well understood, and still competitive when tuned carefully.
- SGD with momentum: Adds a memory of previous updates, so the weights keep moving in a consistent direction instead of zigzagging. This smooths the path through noisy gradients and speeds up convergence.
- Adaptive optimizers: The Adam optimizer and its relatives give every weight its own learning rate, adapted from that weight’s gradient history. They usually work well without much tuning, which made Adam the default choice for deep networks.
Choosing an Optimizer
Start with Adam; it trains a wide range of networks reasonably well on default settings. Reach for tuned SGD with momentum when squeezing out the last bit of performance matters and you have the budget to experiment.
Whatever the optimizer, the learning rate remains the hyperparameter that decides success: too high and training diverges, too low and it crawls.
Example of an Optimizer
Consider training an image classifier. After one mini-batch of photos, the loss says the network mislabeled a quarter of them. Backpropagation computes each weight’s gradient, and the optimizer applies the update: plain SGD steps every weight by the same rule, while Adam gives a rarely-activated filter a bolder step and a frequently-updated one a gentler nudge. Repeated over thousands of batches, those small decisions are what turn random weights into a working classifier.
Related AI terms: Adam Optimizer · Gradient Descent · Stochastic Gradient Descent · Learning Rate · Model Training
Did you like the Optimizer gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About