Gradient Clipping
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A technique that caps gradient size during backpropagation so one oversized update cannot destabilize training. The standard remedy for exploding gradients.
What is Gradient Clipping?
Gradient clipping caps the size of gradients during backpropagation: when a gradient exceeds a chosen threshold, it is scaled down or clamped before the weights update. This stops a single oversized gradient from throwing training into instability.
It is the standard remedy for exploding gradients, the mirror image of the vanishing gradient problem. The same layer-by-layer multiplication that can shrink gradients to nothing can also blow them up when the factors exceed one; clipping addresses only the exploding case.
Methods of Gradient Clipping
- Clipping by norm rescales the whole gradient vector when its length passes a threshold \(c\):
The gradient keeps its direction and loses only its excess magnitude, which is why norm clipping is the usual choice.
- Clipping by value clamps each gradient component individually to a range such as \([-c, c]\). It is simpler, but clipping components independently changes the gradient’s direction, not just its size.
The threshold is a hyperparameter. Set it too low and every update gets truncated, which slows learning; set it too high and it never triggers.
Example of Gradient Clipping
Train a recurrent neural network language model on long documents. Backpropagation through hundreds of time steps multiplies many factors together, and occasionally a rare input sequence produces an enormous gradient.
Without clipping, that one gradient launches the weights far from anything sensible. The loss spikes, sometimes all the way to NaN, and progress up to that point is destroyed.
With the gradient norm clipped at 1.0, a common default, the same spike arrives as an ordinary-sized step in the same direction. Training absorbs the rare event and continues.
Related AI terms: Vanishing Gradient · Backpropagation · Recurrent Neural Network · LSTM · Learning Rate
Did you like the Gradient Clipping gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About