Vanishing Gradient
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A training failure in deep networks where gradients shrink toward zero as they flow backward through the layers, so the earliest layers stop learning.
What is a Vanishing Gradient?
A vanishing gradient occurs when the error signals that train a deep neural network shrink toward zero as they travel backward through its layers, so the layers nearest the input receive almost no signal. Those early layers barely update while later layers keep learning, and the network stalls far short of its potential.
The problem was a central obstacle to training deep networks before the 2010s, and the fixes for it are part of why deep learning works today.
How Vanishing Gradients Happen
Backpropagation computes each layer’s gradient by multiplying together the local derivatives of every layer above it, an application of the chain rule. When those derivatives are smaller than one, the product shrinks exponentially with depth.
The classic culprit is the choice of activation function. The sigmoid function has a derivative that never exceeds 0.25, so ten sigmoid layers multiply the gradient by at most 0.25¹⁰, roughly one millionth.
The mirror image is the exploding gradient: when the multiplied factors exceed one, gradients grow exponentially instead of shrinking. Gradient clipping treats the exploding case; it does nothing for the vanishing one.
How to Fix a Vanishing Gradient
- ReLU activations have a derivative of exactly 1 for positive inputs, so gradients pass through active units undiminished.
- Careful weight initialization (Xavier or He schemes) sets starting weights at a scale that keeps signal magnitudes steady across layers.
- Batch normalization rescales each layer’s inputs during training, keeping activations out of the flat regions where derivatives die.
- Residual (skip) connections give gradients a shortcut past blocks of layers, which is how ResNets train at depths of 100 layers and more. In recurrent networks, LSTM gating serves the same purpose across time steps.
Example of a Vanishing Gradient
Build a 12-layer network with sigmoid activations and train it on image data. The gradient reaching the first layer has passed through twelve derivatives of at most 0.25 each, a factor below 0.25¹², about six in a hundred million.
Multiply any reasonable learning rate by that factor and the first layer’s weights move by effectively nothing per step. Training loss plateaus, the early layers stay close to their random starting point, and the network never learns the basic edge and texture detectors it needs. Swap the sigmoids for ReLU and initialize with the He scheme, and the same architecture trains.
Related AI terms: Gradient Clipping · Weight Initialization · Backpropagation · Activation Function · Sigmoid Function
Did you like the Vanishing Gradient gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About