Explainability
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
The ability to give a person an understandable reason for a specific AI output, usually through post-hoc techniques like SHAP or LIME applied to trained models.
What is Explainability?
Explainability is the ability to give a person an understandable reason for a specific output of an AI system, usually by applying post-hoc analysis to a model whose internals are opaque. The field that builds these techniques is called explainable AI (XAI).
The need is practical: an applicant denied credit, a clinician weighing an AI-suggested diagnosis, and a regulator auditing a deployed system all have to know why the model produced the answer it did, and a black-box model does not volunteer that on its own.
How Explainability Works
Explanation techniques treat the trained model as a fixed object and probe it from the outside.
- Feature attribution scores how much each input contributed to a prediction. SHAP does this with Shapley values borrowed from game theory; LIME does it by fitting a simple surrogate model around a single prediction.
- Counterfactual explanations answer a different question: what is the smallest change to the input that would flip the decision? “Had your income been €4,000 higher, the loan would have been approved.”
- Global explanations describe the model’s overall behavior, such as which features matter most across the whole dataset, rather than accounting for one case.
One caution applies to all of them: an explanation is an approximation of the model, not the model itself. A plausible-sounding explanation can be unfaithful to what the model actually computed.
Explainability vs Interpretability
Interpretability means the model’s inner workings are understandable by design; explainability means producing a post-hoc explanation for the output of a model that is not.
A small decision tree needs no explanation layer, because you read its logic directly. SHAP and LIME exist for the opposite case: models too complex to read, where the explanation has to be reconstructed from outside.
| Explainability | Interpretability | |
|---|---|---|
| Where understanding comes from | Reconstructed after the fact, from outside the model | The model’s own structure, by design |
| Works on | Any model, including black boxes | Simple models: linear, trees, small rule sets |
| Typical tools | SHAP, LIME, counterfactuals | None needed; the logic is the model |
| Main risk | Explanation may not match what the model computed | Simplicity can cost accuracy on complex tasks |
Example of Explainability
Under the US Equal Credit Opportunity Act, a lender that denies credit must tell the applicant the principal reasons for the decision. Suppose a bank scores applications with a gradient-boosted model.
An application is denied, so the bank runs SHAP on that single prediction. The attribution shows that high credit-card utilization and a short credit history pushed the score below the approval threshold, while income pushed it up but not far enough.
Those top negative factors become the stated reasons on the adverse action notice. The model stays complex; the explanation makes one decision legible and legally defensible.
FAQ
Is there a legal right to an explanation of an AI decision?
In some contexts. The GDPR entitles people to meaningful information about the logic behind decisions based solely on automated processing, and the US Equal Credit Opportunity Act requires lenders to state the principal reasons for a denial. Whether the GDPR adds up to a full “right to explanation” is still debated among lawyers.
What is the difference between SHAP and LIME?
Both attribute a prediction to input features, by different routes. SHAP distributes credit using Shapley values from game theory, which gives it consistency guarantees at a higher computational cost; LIME fits a simple surrogate model around one prediction, which is faster but less stable across runs. Either way, the result is an approximation of the model, not a readout of it.
Related AI terms: Interpretability · Transparency · XAI · Black-box Model
Did you like the Explainability gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About