Backpropagation
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The algorithm that trains neural networks by computing how much each weight contributed to the error and passing that signal backward through the layers.
What is Backpropagation?
Backpropagation is the algorithm that trains neural networks: it computes, for every weight in the network, how much a small change to that weight would change the prediction error. Those values, the gradients, tell the optimizer which direction to nudge each weight.
Without it, training a network with millions of weights would be hopeless guesswork. Its popularization in a 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams is a large part of why neural networks work at all.
How Backpropagation Works
Training starts with a forward pass: an input flows through the neural network and produces a prediction, and a loss function \( L \) scores how far that prediction is from the truth.
The backward pass then applies the chain rule of calculus layer by layer, from the output back toward the input. This yields the gradient \( \frac{\partial L}{\partial w} \) for every weight in one sweep, at roughly the cost of a second forward pass. Gradient descent completes the step:
\[ w \leftarrow w - \eta \frac{\partial L}{\partial w} \]One caveat matters in deep networks: gradients shrink as they travel backward through many layers, and when they get too small the early layers stop learning. This is the vanishing gradient problem.
Backpropagation vs Forward Propagation
Forward propagation computes the network’s prediction; backpropagation computes how to correct it. Forward passes run every time the model is used, in training and in production alike, while backpropagation runs only during training. The umbrella entry on propagation covers both directions.
| Criterion | Backpropagation | Forward Propagation |
|---|---|---|
| Direction | Output layer back to input | Input layer to output |
| When it runs | Training only | Every prediction, training or deployed |
| What it computes | Gradient of the loss for each weight | Activations and the final prediction |
| What follows | A weight update | A prediction (and, in training, a loss score) |
Example of Backpropagation
A network learning to read handwritten digits sees an image of a 7 and predicts “1” with high confidence. The loss function scores this as a large error.
Backpropagation traces that error backward. Weights in the output layer that voted for “1” receive strong negative gradients; the trail continues into the hidden layers, assigning each weight its exact share of the blame via the chain rule.
Gradient descent then shifts every weight slightly against its gradient. Shown the same 7 again, the network leans a little closer to the right answer, and millions of repetitions turn that lean into accuracy.
Related AI terms: Propagation · Gradient Descent · Weights · Chain Rule · Vanishing Gradient
Did you like the Backpropagation gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About