Optimization
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The mathematical process of finding values that minimize or maximize an objective; in ML, the adjustment of a model's weights to reduce a loss function.
What is Optimization?
Optimization is the mathematical process of finding the values that minimize or maximize an objective function. In machine learning it is the engine of training: adjusting a model’s weights, step by step, to make a loss function as small as possible.
The everyday sense of the word, making something better in general, is not what this page describes. Here optimization has an explicit objective, a defined set of adjustable values, and an algorithm that moves between them.
How Optimization Works in Model Training
Training starts with a loss function that measures how wrong the model’s predictions are. Gradient descent computes, for every weight, which direction reduces that loss, and nudges each weight that way. Repeat thousands or millions of times and the loss falls.
Variants of this idea dominate practice. Stochastic gradient descent estimates the gradient from a small batch of data instead of the whole dataset, and the Adam optimizer adapts the step size for each weight. In deep learning the loss landscape is non-convex, full of valleys and saddle points, so these methods find good solutions rather than provably perfect ones.
Optimization vs Hyperparameter Optimization
The practical difference: model optimization adjusts weights inside a single training run, while hyperparameter optimization, better known as hyperparameter tuning, searches across many training runs for the best settings to launch them with.
| Model optimization | Hyperparameter tuning | |
|---|---|---|
| What changes | Weights and biases | Settings like learning rate or depth |
| What changes it | Gradient descent within one run | A search loop across training runs |
| Guided by | Loss on training data | Score on validation data |
| Scale | Thousands to millions of weight updates | Tens to hundreds of trials |
Example of Optimization
An engineer trains an image classifier. The loss is cross-entropy: large when the model assigns low probability to the correct label.
Each training step feeds a batch of images through the network, measures the loss, and computes the gradient of that loss with respect to every weight. The optimizer moves each weight a small step downhill. After enough steps the loss curve flattens near a minimum, and the model labels most images correctly.
Related AI terms: Gradient Descent · Adam Optimizer · Hyperparameter Tuning · Scheduler · Loss Landscape
Did you like the Optimization gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About