Information Gain
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A measure of how much a feature reduces uncertainty (entropy) when data is split on it, and the criterion decision trees use to choose each split.
What is Information Gain?
Information gain measures how much uncertainty about the target drops when a dataset is split on a feature. It is the entropy of the data before the split minus the weighted entropy of the pieces afterward; the bigger the drop, the more the feature tells you.
It is best known as the splitting criterion in decision trees: at every node, the tree computes the information gain of each candidate feature and splits on the winner.
How Information Gain Works
Entropy quantifies impurity. A group where every example has the same label has entropy zero; a 50/50 mix has the maximum.
To score a split, take the entropy of the parent set, split the data on the feature, and compute the entropy of each child group weighted by its share of the examples. Information gain is the parent entropy minus that weighted sum.
One caveat: raw information gain favors features with many distinct values, because an ID-like column splits the data into perfectly pure single-row groups. The gain-ratio variant corrects for this by penalizing splits with many branches.
Applications of Information Gain
- Decision tree splits: ID3 and C4.5, two classic tree-building algorithms, choose each split this way (C4.5 uses the gain ratio).
- Feature selection: As a filter method, features are ranked by information gain against the target, and low scorers are dropped before training.
- Text classification: Selecting the words whose presence says the most about a document’s label.
Example of Information Gain
A retailer builds a decision tree to predict whether a visitor buys, using age group and income bracket. Before any split, the training set is 50% buyers: maximum entropy.
Splitting on income produces groups that are each still roughly half buyers. Entropy barely moves, so information gain is near zero. Splitting on age produces one group that is 80% buyers and another that is 15% buyers.
The age split cuts entropy far more, so its information gain is higher, and the tree splits on age first. The process repeats inside each branch until the leaves are pure enough or no split helps.
Related AI terms: Decision Tree · Feature Selection · Random Forest · Cross-Entropy Loss
Did you like the Information Gain gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About