Validity
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The degree to which an AI system actually measures or predicts what it claims to; paired with reliability in NIST's definition of trustworthy AI.
What is Validity?
Validity is the degree to which an AI system measures or predicts what it claims to, rather than some proxy that merely correlates with it.
The NIST AI Risk Management Framework lists “valid and reliable” as the first characteristic of trustworthy AI, ahead of safety, security, and fairness. The pairing is deliberate: a system can produce consistent output that consistently measures the wrong thing.
Types of Validity
- Construct validity: the system measures the concept it was built for. A hiring model that claims to predict job performance but actually predicts resemblance to past hires fails here.
- Internal validity: the results reflect what was tested, not an artifact of the test itself, such as data leakage between training and test sets.
- External validity: the results generalize beyond the evaluation data to the population and conditions the system will face in deployment, and keep holding as the world shifts.
Validity vs Reliability
Validity asks whether the system measures the right thing; reliability asks whether it performs consistently over time. The two fail independently, which is why NIST demands both.
A credit model that scores the same applicant identically every day is reliable; if the score tracks postal codes rather than repayment risk, it is not valid. Model evaluation against trusted ground truth probes validity; sustained monitoring in operation probes reliability. A serious AI assessment gathers evidence for both.
Example of Validity
A 2019 study in Science by Ziad Obermeyer and colleagues examined a widely used US healthcare algorithm that scored patients for extra-care programs. The system was treated as a measure of medical need, but it was trained to predict healthcare costs.
Cost is not need. Less money was spent on Black patients than on equally sick white patients, so at the same risk score, Black patients were sicker, and fewer qualified for the extra care they needed.
Nothing about the system was inconsistent. It predicted costs accurately and repeatably; the failure was validity, a proxy target quietly standing in for the thing that mattered.
Related AI terms: Reliability · Model Evaluation · AI Assessment · Ground Truth · Trustworthy AI
Did you like the Validity gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About