Weight Decay
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A regularization method that shrinks weights toward zero at each training step to curb overfitting. Equivalent to L2 regularization under plain SGD.
What is Weight Decay?
Weight decay is a regularization method that shrinks a model’s weights toward zero at every training step, penalizing large weights in proportion to their size. Smaller weights mean a smoother, simpler function, which reduces overfitting.
Under plain stochastic gradient descent, weight decay is mathematically equivalent to L2 regularization: adding \(\frac{\lambda}{2}\lVert w \rVert^2\) to the loss produces exactly the same update as decaying the weights directly.
How Weight Decay Works
Each update subtracts a fraction of every weight alongside the usual gradient step:
\[ w \leftarrow w - \eta \left( \nabla L(w) + \lambda w \right) \]where \(\eta\) is the learning rate and \(\lambda\) the decay coefficient, a hyperparameter. Rearranged, each step multiplies the weights by \((1 - \eta\lambda)\) before applying the gradient, hence “decay.”
The effect is a constant pull toward zero that the data must argue against. Weights that genuinely reduce the loss survive the pull; weights that only memorize noise in the training set drift back toward zero.
With adaptive optimizers, the L2 equivalence breaks: the penalty gradient gets rescaled by the Adam optimizer’s per-parameter step sizes, which weakens the regularization exactly where updates are largest. AdamW fixes this by applying the decay directly to the weights, decoupled from the gradient, and has become the standard variant for training transformers.
Example of Weight Decay
A house-price model with 300 features overfits: it assigns large weights to spurious features that correlate with price only in the training data, scores well there, and misses badly on new listings.
Retrain it with weight decay at \(\lambda = 0.01\). The constant pull toward zero costs the spurious features their large weights, because the small loss improvement they offer no longer outweighs the penalty. Genuinely predictive features like square footage keep substantial weights, since shrinking them would raise the loss more than the penalty saves.
Training error rises slightly; error on unseen listings drops. That trade is the whole point of regularization.
Related AI terms: Regularization · Overfitting · Adam Optimizer · Loss Landscape · Hyperparameter
Did you like the Weight Decay gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About