Hypothesis Testing
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
A statistical method for deciding whether sample data supports a claim about a population, and the machinery behind A/B tests of models and products.
What is Hypothesis Testing?
Hypothesis testing is a statistical method for deciding whether sample data supports a claim about a population, by weighing the evidence against a default assumption called the null hypothesis.
The structure guards against wishful thinking. Instead of asking “does my change work?”, you assume it does nothing (the null hypothesis, H0) and check whether the observed data would be surprising under that assumption. Only strong evidence justifies accepting the alternative hypothesis (H1).
How Hypothesis Testing Works
The procedure runs in four steps. State the null and alternative hypotheses; collect sample data; compute a test statistic that summarizes how far the data departs from what H0 predicts; and convert that statistic into a p-value, the probability of seeing data at least this extreme if H0 were true.
If the p-value falls below a significance level chosen in advance (0.05 is the convention), you reject the null hypothesis. Which test statistic to use depends on the situation: a z-test when samples are large and population variance is known, a t-test when samples are small, a chi-square test for associations between categorical variables. The z-test gets its name from the z-score it computes, and its logic rests on the normal distribution.
In machine learning, hypothesis testing is how teams tell a real improvement from noise – whether in model evaluation (is the new model’s accuracy gain significant?) or in production A/B tests.
Example of Hypothesis Testing
An e-commerce team builds a new recommendation model and wants to know if it beats the current one. They set up an A/B test: H0 says the new model produces the same click-through rate as the old one, H1 says the rates differ.
For two weeks, incoming users are randomly split, half served by each model. Suppose the old model gets a 3.1% click-through rate and the new one 3.4%. Is that 0.3-point gap real or luck?
The team runs a two-proportion z-test on the results. If the p-value comes in below 0.05, they reject H0, conclude the new model genuinely performs better, and roll it out. If not, the gap is within what random variation would produce, and the old model stays.
Related AI terms: Z-Score · Normal Distribution · Monte Carlo · Model Evaluation
Did you like the Hypothesis Testing gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About