Feature Selection
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The process of choosing the subset of input variables that carries real signal, cutting noise, training cost, and overfitting in machine learning.
What is Feature Selection?
Feature selection is the process of choosing which input variables (features) a machine learning model trains on, keeping the ones that carry signal and dropping the rest. Fewer, better features mean faster training, less overfitting, and a model that is easier to explain.
More data is not always better data. A dataset with thousands of columns tends to contain a handful that matter and a long tail of noise that a model will happily memorize.
Types of Feature Selection
- Filter methods rank features with a statistical score, such as correlation with the target, a chi-square test, or information gain, before any model is trained.
- Wrapper methods train the actual model on candidate feature subsets and keep the subset that performs best, as in forward selection or backward elimination.
- Embedded methods select features as a side effect of training itself; Lasso regression, which shrinks the weights of useless features to zero, is the standard example.
Feature Selection vs Feature Engineering
Feature selection chooses among features that already exist; feature engineering creates and transforms them in the first place. Engineering comes earlier in the pipeline and expands the feature set; selection then prunes it to what earns its place.
| Criterion | Feature selection | Feature engineering |
|---|---|---|
| What it does | Keeps a subset of existing features | Creates or transforms features |
| Input | A finished feature set | Raw or messy data |
| Output | Fewer columns | New or modified columns |
| Typical failure | Discarding a feature that carried signal | Leaking target information into a feature |
Example of Feature Selection
A hospital team predicts a rare disease from gene-expression data: 20,000 measured genes per patient, a few hundred patients. Training on everything guarantees overfitting – the model has far more knobs than examples.
A filter method first cuts the genes to the 200 most correlated with the diagnosis. A wrapper method then searches within those 200 and lands on a panel of 12 genes that predicts as well as the full set.
The 12-gene model is cheaper to run, and clinicians can inspect it: each selected gene becomes a candidate biomarker worth investigating.
Related AI terms: Feature Engineering · Wrapper Method · Information Gain · Dimensionality Reduction · Overfitting
Did you like the Feature Selection gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About