Weight Initialization
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The choice of starting values for a neural network's weights. Set them at the wrong scale and gradients vanish or explode before learning begins.
What is Weight Initialization?
Weight initialization is the choice of starting values for a neural network’s weights before training begins. It matters more than “pick random numbers” suggests: the starting scale decides whether signals and gradients stay usable as they pass through the layers, or vanish and explode before learning starts.
A network initialized well converges quickly. The same network initialized badly trains slowly or not at all.
Common Weight Initialization Methods
- Zero (or constant) initialization fails outright. Every neuron in a layer computes the same output and receives the same gradient, so they all learn identical features and the layer collapses to one neuron’s capacity.
- Small random values break that symmetry, but an arbitrary scale compounds across depth: signals shrink or grow layer by layer until gradients vanish or explode.
- Xavier (Glorot) initialization draws random weights with variance tuned to the number of inputs and outputs of each layer, keeping signal variance roughly constant across depth. It suits sigmoid and tanh activation functions.
- He initialization adapts the same idea to ReLU, which zeroes half its inputs, by doubling the variance:
where \(n_{in}\) is the number of inputs to the layer.
Example of Weight Initialization
Consider a 50-layer ReLU network where each layer, because of its weight scale, multiplies the typical signal magnitude by a constant factor. If that factor is 1.1, the signal grows by 1.1⁵⁰, over a hundredfold, by the top of the network. If it is 0.9, the signal shrinks by 0.9⁵⁰, to about half a percent of its original size.
Neither network can train: one saturates and produces exploding gradients, the other starves its layers of signal and hits a vanishing gradient. He initialization picks the weight variance that holds the factor at one, so the fiftieth layer receives a signal as healthy as the first.
The lesson generalizes. Depth amplifies any per-layer bias in scale exponentially, so initialization schemes exist to make that per-layer factor exactly neutral.
Related AI terms: Vanishing Gradient · Weights · Activation Function · Backpropagation · Gradient Clipping
Did you like the Weight Initialization gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About