Softmax
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A function that converts a vector of raw scores into probabilities summing to 1, used as the output layer in multi-class classification networks.
What is Softmax?
Softmax converts a vector of raw scores (logits) into probabilities that are all positive and sum to 1, so a model’s output reads as a probability distribution over classes.
It is the standard final layer wherever a network must choose one answer among several: which digit, which language, which word comes next. The survey entry on the activation function places it among its peers; this page looks closer.
How Softmax Works
For a vector of scores \( z \), the probability assigned to class \( i \) is:
\[ \text{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j} e^{z_j}} \]Exponentiation makes every score positive and stretches the gaps between them, so the leader pulls ahead; division by the sum forces the results to total exactly 1. The ranking of the scores survives untouched: the top logit always becomes the top probability.
During training, softmax pairs naturally with cross-entropy loss, which punishes the network for assigning low probability to the true class. The same mechanism sits inside every large language model: at each step, a transformer applies softmax over its entire vocabulary to decide which token comes next.
Softmax vs Sigmoid
Softmax produces a set of probabilities that compete and sum to 1; the sigmoid function produces one independent probability. Use softmax when exactly one class must win, and sigmoid when the question is yes-or-no or when several labels can be true at once.
For two classes the pair coincide: softmax over two scores equals a sigmoid of their difference.
| Softmax | Sigmoid | |
|---|---|---|
| Input | A vector of scores | A single score |
| Output | Probabilities that sum to 1 | One probability between 0 and 1 |
| Task | Single-choice multi-class classification | Binary or multi-label classification |
| Independence | Raising one class’s probability lowers the others | Each output stands alone |
Example of Softmax
An image classification network scores a photo against three classes – cat, dog, fox – and produces the logits \([2.0, 1.0, 0.1]\). Softmax exponentiates each score and divides by the total:
\[ \text{Softmax}(2.0) = \frac{e^{2.0}}{e^{2.0} + e^{1.0} + e^{0.1}} \approx 0.659 \]The full result is \([0.659, 0.242, 0.099]\): 66% cat, 24% dog, 10% fox. Note what the exponentials did to the gaps. Cat scored twice dog’s logit but receives nearly three times its probability, and the three numbers now sum to 1, so the network’s answer is a genuine probability distribution rather than a raw score.
Related AI terms: Sigmoid Function · Activation Function · Cross-Entropy Loss · Classification · NLP Transformer
Did you like the Softmax gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About