Joint Probability Distribution

How a Joint Probability Distribution Works

Three quantities derive from the joint distribution, and together they answer most probability questions:

1. Marginal probability: The probability of one variable alone, obtained by summing the joint distribution over the other variable’s values: \(P(X) = \sum_{y} P(X, y)\).
2. Conditional probability: The probability of one event given that another occurred: \(P(X \mid Y) = \frac{P(X, Y)}{P(Y)}\).
3. Independence: Two variables are independent exactly when the joint distribution factors into the product of the marginals: \(P(X, Y) = P(X) \cdot P(Y)\). When it does not, knowing one variable tells you something about the other.

This is the object much of machine learning tries to capture. A Bayesian network is a compact encoding of a joint distribution over many variables, Naive Bayes classifies by assuming the joint distribution factorizes, and generative models learn a joint distribution over data so they can sample new examples from it.

Example of a Joint Probability Distribution

Roll a fair six-sided die (\(X\)) and toss a fair coin (\(Y\)). The die and coin do not influence each other, so the joint probability of any combination is the product of the individual probabilities:

\[ P(X = x, Y = y) = P(X = x) \cdot P(Y = y) = \frac{1}{6} \times \frac{1}{2} = \frac{1}{12} \]

Each of the 12 combinations, from (1, Heads) through (6, Tails), has probability \(\frac{1}{12}\), and the twelve values sum to 1, as any probability distribution must.

Now replace the coin with a variable that depends on the die, say \(Y\) = “roll is even”. The factorization breaks: \(P(X = 2, Y = \text{yes}) = \frac{1}{6}\), not \(\frac{1}{6} \times \frac{1}{2}\). The joint distribution is where that dependence becomes visible.

Related AI terms: Bayesian Network · Naive Bayes · Normal Distribution · Multivariate Analysis

Did you like the Joint Probability Distribution gist?

Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.