AIG-017 AI Model Explainability
Description
For AI systems that produce decisions or recommendations affecting users, documentation of the model's explainability capabilities exists. This includes: the type and level of explanation available (feature attribution, confidence scores, decision paths), known limits on generalisability, and guidance for operators on how to interpret outputs. Where explanations are not technically feasible, this limitation is documented, disclosed to users, and factored into human oversight design.
Rationale
Unexplainable AI outputs prevent operators from identifying errors, challenging decisions, or understanding when to override; explainability is also a legal requirement for certain automated decisions.
Applicability (9 profiles)
Obtain the explainability documentation from the provider (AIG-015). The deployer uses it in the AIG-022 oversight design and the AIG-016 explanation route.
Art.13(3) names the explainability capabilities as mandatory content of the instructions for use, which gives this documentation a fixed audience and a fixed home: it is written for the deployer, inside the instructions, next to the performance metrics and the guidance on interpreting outputs. Art.13(1) is the standard the result is judged against, operation transparent enough for the deployer to interpret an output and use it appropriately. Where explanation is not technically feasible, the recorded limitation is what the instructions have to state rather than something held internally.
Carried from the base. The explainability documentation comes from the provider under AIG-015 and this profile uses it twice: it is what the Art.86(1) explanation is built from and it is the evidence behind the human oversight element the Art.27 seat has to describe at point (e) of its assessment. The first use binds both seats and the second only the Art.27 one.
Framework Mappings (13)
| GRC-13 | Explainability Requirement | partial |
| GRC-14 | Explainability Evaluation | full |
| EU-AI-Art.13.1 | Transparency — System Transparency for Deployers | partial |
| EU-AI-Art.13.3 | Transparency — Mandatory Content of Instructions for Use | informative |
| EU-AI-Art.86 | Deployer Obligations — Right to Explanation of Individual Decision-Making | partial |
| GDPR-Art.22 | Automated Decision-Making and Profiling | informative |
| GV-4.1-001 | Safety-First Organisational Culture | GV-4.1-001 | full |
| MG-3.2-001 | Pre-Trained Model Monitoring | MG-3.2-001 | full |
| MG-4.2-003 | Continual Improvement Integration | MG-4.2-003 | partial |
| MS-2.9-001 | AI Model Explainability and Validation | MS-2.9-001 | full |
| MS-4.2-003 | Trustworthiness Measurement with Expert Input | MS-4.2-003 | full |
| MAP 2.2 | AI System Knowledge Limits Documentation | full |
| MEASURE 2.9 | AI Model Explainability and Validation | full |
Evidence (2)
Model explainability documentation describing the explanation type and level available (feature attribution, confidence scores, decision paths), known limits, and operator guidance for interpreting outputs.
Example: Model Card · Credit Risk Model v2 (MLflow model card), section 'Explainability': SHAP feature importance explanations available via API, confidence score returned with each prediction, documented precision at confidence thresholds, and operator guidance on interpreting low-confidence outputs
Test: Request explainability documentation for AI systems making decisions affecting users. Verify: (1) explanation type and level are specified, (2) known limits on generalisation are stated, (3) operator guidance for interpreting outputs is included, (4) where explanations are not technically feasible, this limitation is explicitly stated, disclosed in user-facing documentation, and factored into the human oversight design for that system.
Export of the explanations the system produced for a sample of decisions, showing the explanation type available to the operator.
Example: Explanation export, credit-decision service, 2026-07-01 to 2026-07-31: 250 sampled decisions, each with feature attributions and a confidence score
Test: Export the explanation returned with each decision for a sample drawn across the period. Verify: (1) every sampled decision carries an explanation of the type the documentation states is available, (2) the explanation resolves to the model version that produced the decision, (3) decisions where no explanation was produced are present in the export with the reason, (4) the explanation content is available to the operator at the point of review rather than only through a later request, (5) the sample covers each decision category the system produces rather than the most common one.
Questions (2)
Is the explainability capability of each AI system that produces decisions or recommendations affecting users documented?
Unexplainable AI outputs prevent operators from identifying errors or challenging decisions. Documentation should specify what explanations are available (feature attribution, confidence scores, decision paths) and, where explanation is not technically feasible, state this limitation explicitly.
Which of the following does your explainability documentation record?
Options run from the most commonly recorded to the least. At least one explanation type applies to any system in scope. Where explanation is not technically feasible the last item is the honest answer. It is a weak position unless the oversight design in AIG-022 carries the weight instead.