Hyperparameter
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A configuration value set before training begins, such as learning rate or batch size, that controls how a model learns, unlike weights learned from data.
What is a Hyperparameter?
A hyperparameter is a configuration value set before training begins, such as the learning rate, network depth, or batch size, that controls how a machine learning model learns.
The word invites confusion with the model’s parameters, which are a different thing entirely: parameters are learned, hyperparameters are chosen. Choose badly and the same data and architecture produce a model that never converges, or one that memorizes instead of generalizing.
Hyperparameters vs Parameters
The practical difference: an engineer picks hyperparameters before training starts, while the training process itself learns the parameters, the weights and biases inside the model. A neural network with millions of weights might have only a dozen hyperparameters, but those dozen decide whether the millions end up anywhere useful.
| Hyperparameter | Parameter (weight) | |
|---|---|---|
| Set by | The engineer, before training | The training algorithm, from data |
| Examples | Learning rate, batch size, tree depth | Network weights, regression coefficients |
| Found through | Hyperparameter tuning | Gradient descent or another optimization method |
| How many | A handful to a few dozen | Thousands to billions |
Types of Hyperparameters
- Architecture hyperparameters define the model’s structure: the number of layers and units in a neural network, or the number of trees in a random forest.
- Algorithm hyperparameters control how learning proceeds: the learning rate, the regularization strength, the choice of optimizer.
- Training hyperparameters govern the loop itself: batch size, number of epochs, and the criteria for early stopping.
Example of a Hyperparameter
The learning rate is the classic case. It fixes how far the weights move on each update during model training.
Set it too high and training overshoots: the loss bounces around and never settles. Set it too low and training crawls, or stalls in a poor solution long before reaching a good one. One number, chosen before the first weight update, separates a model that works from one that doesn’t.
Related AI terms: Hyperparameter Tuning · Grid Search · Weights · Learning Rate · Optimization
Did you like the Hyperparameter gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About