When documented AI failures meet a frozen non-compensatory gate set

Executive overview

Problem: Policy documents describe trustworthy AI principles, but teams still lack transparent, replayable rules for “stop vs go” that do not let one strong score mask a failing domain.

Why it matters: After high-profile system failures, leaders need to know whether a declared governance rule set would have blocked deployment ex ante—not a narrative after the fact. Historical replay answers that question for a fixed, pre-specified engine.

What was done

Researchers encoded twelve well-documented failure cases (2014–2021) across seven sectors using a structured supply-chain-style feature model with explicit provenance classes (direct, derived, imputed, uncertain). A versioned overlay reconciles “core-equivalent” legacy rows to expanded evidence. A frozen five-gate engine (four threshold profiles, including “moderate”) evaluates each case; a parallel compensatory comparator scores the same evidence with offsetting allowed.

The study adds regulatory-cleared control devices, expanded benchmark tiers, Monte Carlo bands on confidence, perturbation sweeps, dual-schema invariance checks, and single- versus pairwise-gate ablations to test whether rejections are fragile artefacts of one gate.

What was found

Under the moderate profile on the twelve core failures, eleven are rejected and one is a false negative (narrow safety margin). All twelve control devices approve, supporting specificity. The safety gate binds most often; bias, calibration, and traceability failures appear repeatedly; evidence depth binds less often but matters in ablations.

Layer discipline: headline confusion-matrix statistics for the twelve-failure presentation do not match the single exported confusion-matrix file, which encodes a different evaluation layer (expanded failures plus controls). Both are documented; mixing them invalidates interpretation (P4-C10). One quantity in the sensitivity narrative shows a small manuscript–log discrepancy; the traceability row for P4-C21 preserves the engineer flag.

Removing any one gate never flips a rejection; selected pairs do—showing conjunctive structure. Compensatory scoring agrees on most failures but would approve two notable cases the non-compensatory engine rejects. Expanded benchmarks can show perfect separation under the stated encoding—a structural observation, not a claim of prospective predictive validation (P4-C25).

Why this matters for regulation, safety, and deployment

• Procurement and assurance: buyers can demand replayable gate evidence instead of narrative compliance alone.
• Risk committees: conjunctive rules surface multi-domain weaknesses early; compensatory models may hide them.
• Post-incident review: the same frozen engine can be applied consistently across cases for audit defensibility.

Limitations and ethics

Convenience sample; heterogeneous provenance; selection and survivorship bias; not representative of all jurisdictions or deployments. Retrospective computational study on public materials only—no human subjects. Assistive tools may support documentation; scientific judgments remain author-owned (P4-C35).