Methodology & Scoring
1. Our approach
The Workbench's analytical tools (System Intake, Fairness Scanner, Drift & Data Quality, Doc Gap Scanner,
the readiness score) are all deterministic rule engines written in plain JavaScript
(engines/risk.js, engines/fairness.js, engines/drift.js,
engines/remediation.js) — the same inputs always produce the same outputs, and every output
carries a human-readable reason. None of these five tools call a language model. Separately, a handful of
features are explicitly AI-assisted (the Governance Copilot, Agent Loop, and the blog) — see §7 for exactly
which ones and how that differs.
2. Risk-tier classification
The System Intake tool maps your answers to an EU AI Act risk tier — prohibited, high, limited, or minimal — using a fixed decision table, not a probabilistic model:
- Prohibited — you flagged a practice listed under EU AI Act Art. 5 (e.g. social scoring, subliminal manipulation, untargeted biometric scraping).
- High — the system operates in one of the eleven Annex III high-risk domains (employment, credit/finance, insurance, healthcare, essential services, critical infrastructure, biometric ID, law enforcement, migration, justice, education) and materially affects a decision about a person.
- Limited — the system uses generative AI or interacts directly with people (e.g. a chatbot), triggering the Art. 50 transparency duty, without meeting the High-risk criteria.
- Minimal — none of the above conditions are met.
Every classification returns a reasons[] list stating exactly which answer(s) drove the outcome —
shown to you in the tool itself, not hidden.
3. Readiness score (0–100)
The readiness score sums four independently-capped components — it is not a machine-learning prediction, just arithmetic:
| Component | Max points | How it's computed |
|---|---|---|
| Documentation completeness | 35 | (Doc Gap Scanner completeness % ÷ 100) × 35. If you haven't run the scanner yet, a flat 8-point placeholder is used instead of assuming full or zero coverage. |
| Fairness | 25 | 25 if the Fairness Scanner passes; otherwise 25 − (8 × number of failing groups), floored at 0. If you haven't run it yet: a flat 12-point placeholder. |
| Drift & data quality | 20 | 20 − (5 × number of drift/quality flags), floored at 0. If neither tool has been run: a flat 9-point placeholder. |
| Remaining process burden by tier | 20 | A fixed allowance reflecting how much obligation work is inherent to your tier: Prohibited 0, High 8, Limited 14, Minimal 20. |
The four components are summed and rounded to the nearest whole number, then clamped to 0–100. If your system is classified Prohibited, the score is additionally capped at 25 regardless of the other components — a prohibited practice cannot be scored as "ready." The resulting number maps to a band: 80+ Strong, 60–79 Developing, 35–59 Early, below 35 At risk.
4. Governance Effectiveness Score (0–100)
A separate, self-assessed score from the readiness score above: you supply the eight inputs
yourself at /effectiveness-score, and nothing you enter is verified, stored,
or sent to us. Where the readiness score rates one assessment, this one is about an organization's governance
as a whole. Current formula version: G v1.1.
Every table in this section is rendered directly from engines/effectiveness.js — the same module
the calculator runs — so the published methodology cannot drift out of sync with the arithmetic.
G = 100 × C × (V^0.15 × O^0.20 × K^0.25 × E^0.15 × R^0.15 × I^0.10) × (1 − P)
The six bracketed exponents sum to exactly 1.00, making the bracket a weighted geometric mean. C and P enter linearly and are by far the most powerful terms — a ten-point change in coverage moves the score roughly six times as far as a ten-point change in evidence.
| Factor | Weight | What it measures | How to measure it |
|---|---|---|---|
| C — Coverage | linear × |
How much of your critical systems, data, vendors, processes, and AI use cases are inside governance. | Governed AI assets divided by every AI asset you can discover, shadow AI included. Count systems, models, agents, datasets and vendors — not documents. |
| V — Visibility | ^0.15 |
Can you see important access, actions, changes, and decisions? | Share of production AI systems that emit a reviewable log of access, actions, changes and decisions, over the last 90 days. |
| O — Ownership | ^0.20 |
Does every important risk, control, exception, and decision have a named owner? | Share of open risks, controls, exceptions and governance decisions with one named accountable person. A team name or a vacant role counts as zero. |
| K — Key Control Reliability | ^0.25 |
Do your most important controls work when they are tested? | Pass rate of your key controls on their last test, counting any control past its retest date as a failure rather than as untested. |
| E — Evidence | ^0.15 |
Can you prove your controls and governance processes are actually working? | Share of key controls whose last passing result is inside its evidence-freshness window and retrievable on demand without asking a person. |
| R — Response | ^0.15 |
Are serious issues contained and resolved within the required time? | Share of critical and major issues closed within their resolution target over the last 90 days — 72 hours for critical, 14 days for major, until you set your own. |
| I — Improvement | ^0.10 |
Are you preventing repeated failures, or simply recording them again? | One minus the repeat-failure rate: of the controls and incidents that failed and were remediated in the last 12 months, the share that failed again the same way. |
| P — Critical Exposure Penalty | linear − |
How much overdue and unaccepted risk currently sits above your approved risk level. | Share of your AI systems carrying at least one risk exception or acceptance that is past its expiry or review date, weighted by each system’s risk tier. |
The weakest-factor rule
Multiplication alone does not deliver the "a strong factor cannot hide a weak one" claim this score is built
on. A geometric mean with fractional exponents is a gentle averager: with every other factor at 100, key
control reliability at 10% still scored 56, and improvement at 10% still scored 79. Since
G v1.1 the score is therefore capped:
| Score cannot exceed | When any factor is | Or the exposure penalty is |
|---|---|---|
| 49 | at or below 20% | at or above 30% |
| 64 | at or below 40% | at or above 20% |
Thresholds are inclusive — a factor at exactly 40% triggers the cap. When a cap applies, the calculator shows the uncapped value and names the factor that caused it, rather than quietly showing a lower number. The alternative mechanism considered — blending the minimum factor into the geometric mean — was rejected because it measurably inflated mid-range scores: the neutral all-75s case rose from 50.6 to 56.0, which is the opposite of what a weakest-link mechanism should do.
Bands
| Score | Band | What it means |
|---|---|---|
| 80–100 | Strong — supported by evidence | Governance works and can be proven. An auditor, regulator, or customer could be shown the working, not just the intent. |
| 65–79 | Working — important gaps remain | The basics hold, but at least one factor is weak enough to undermine the others. Fix the weakest factor before adding new governance activity. |
| 50–64 | Weak — corrective action required | Governance exists on paper and partly in practice, but the organization cannot reliably prove control. Expect audit findings and avoidable surprises. |
| Below 50 | Serious unmanaged exposure | Risk is being carried without visibility, ownership, or proof. This is an active exposure, not a documentation backlog. |
These band boundaries are not yet calibrated against outcomes. They are reasoned cut points, not empirically derived ones, and we would rather say so than imply a precision the data does not support.
How recommendations are ranked
The calculator ranks what to fix by what each move is actually worth: for every factor it
recomputes the whole score with that factor improved by 10 points and ranks by the
resulting gain. This is deliberately not the same as ranking by which factor looks weakest — on the
calculator's own worked example those two orders disagree, with the weakest-first ordering recommending a move
worth 1.05 points ahead of one worth 6.34. Versions before G v1.1 ranked by
weakness and were wrong to.
Measured mode, and what it does when data is missing
The calculator on the public page is an estimate. Signed-in organizations get the
measured version from their account page: the same eight factors, computed by
engines/effectiveness-measure.js from their own records — the AI registry and inventory, the
Controls Register, the Control Plane ledger, governance decisions, risk acceptances, policy exceptions, the
internal incident log and remediation history. Every factor opens the rows behind it.
Three limits are stated rather than papered over:
- Coverage cannot see AI it was never told about. Shadow-AI detection runs in the browser and stores no findings, so there is no denominator of undiscovered systems. Measured coverage is the share of the registered estate that is governed — an upper bound on real coverage, not a measurement of it.
- Response measures time to close, not time to contain, because the incident record has a detection and a closure timestamp but no containment one. Default targets are 3 days for a critical issue and 14 for a major one, until an organization sets its own.
- A factor with no data reports "Not measured" and is dropped, with the remaining exponents renormalized to sum to 1 — never defaulted to a flattering value. Below four measurable factors, no score is produced at all, because a G computed from two factors is not a G.
Signed attestations
A measured score can be published as a signed attestation at /effectiveness/:id, verifiable by
anyone with no account, using the same ECDSA P-256 key as Trust Passports, evidence packages, AI-BOMs,
assurance reports and diligence reports. The signed payload carries the factor values, the measurement
definition for each one, and the formula version — an attestation that reported a number without saying how it
was measured would be a signed opinion. Attestations can be revoked, and a revoked one says so rather than
disappearing.
Cohort benchmarking
Opt-in and off by default. A percentile is only computed from other organizations that have also opted in, and only when the cohort reaches at least 8 organizations — below that the platform reports why it will not show a number instead of showing a soft one. Percentiles are rounded to the nearest 5 points and the spread is reported as quartiles, because a minimum or maximum in a small cohort is one organization's score.
Calibration — not yet claimed
The band boundaries above are reasoned cut points, not empirically derived ones, and this page will keep saying so until that changes. The machinery to test the claim is built: the platform records each score with a timestamp and can count the serious incidents an organization went on to have afterwards, so the question "do lower scores actually predict more incidents?" is directly answerable. It will not be answered publicly until the sample supports it — at least 30 organizations and 50 observed incidents — and when it is, the correlation will be published with its confidence interval, its per-factor breakdown, and whatever it kills. If a factor turns out to predict nothing, dropping it in public is worth more than the factor was.
Changelog
| Version | Change |
|---|---|
G v1.0 | Launch formula. No cap rule; recommendations ranked by raw weakness. |
G v1.1 | Added the weakest-factor cap rule; recommendations now ranked by measured point gain; a measurement definition published for every factor; the score explicitly labelled a self-assessed estimate. |
5. Fairness Scanner
The scanner computes up to four independent group-fairness metrics for every protected attribute in your data — independent in the sense that a dataset can pass one and still fail another, because each answers a different question about what "fair" means. This is not this platform's own taxonomy: it mirrors the three classical fairness criteria the research literature and IBM's open-source AIF360 toolkit organize group-fairness metrics around (Independence, Separation, Sufficiency), plus the disparate-impact screening ratio regulators actually use.
| Metric | Criterion | What it measures | Default threshold |
|---|---|---|---|
| Disparate impact ratio | Independence | Each group's selection rate ÷ the reference group's rate. | Fails below 0.80 (the four-fifths rule) |
| Demographic parity difference | Independence | Gap between the highest and lowest group selection rates. | Flags above 10 percentage points |
| Equal opportunity difference | Separation | Gap in true-positive rate (of people who should have been selected, how many were) across groups. Requires a ground-truth label column. | Flags above 10pp |
| Equalized odds difference | Separation | The worse of the true-positive-rate gap and the false-positive-rate gap across groups. Requires labels. | Flags above 10pp |
| Predictive parity difference | Sufficiency | Gap in positive predictive value (of people the model selected, how many were actually right) across groups. Requires labels. A model can score a perfect 0pp on both Separation metrics above and still fail this one if groups have different underlying base rates — it is checking something those metrics structurally cannot see. | Flags above 10pp |
Every threshold above is configurable per run; 0.80 and 10pp are conventional starting points (the four-fifths rule is a regulatory screening heuristic, not a universal legal determination — see Terms §3), not a claim that your specific sector or regulator uses exactly these numbers. The tool passes only if every computed metric clears its threshold; each failing metric is reported with its exact value and which group(s) drove it.
6. Drift & Data Quality
For numeric features, drift is measured with the Population Stability Index (PSI) computed across 10 quantile bins derived from your baseline dataset; categorical features use a PSI variant across the union of observed categories. A feature is flagged as significant drift when PSI ≥ 0.20 — the conventional industry threshold. Data-quality checks separately flag missing values, schema changes, and outlier concentrations using standard IQR-based bounds.
7. Doc Gap Scanner
Completeness is ((fully-present controls + 0.5 × partially-present controls) ÷ total controls) × 100,
rounded to the nearest whole percent, against a fixed checklist of documentation controls (intended use,
training data, evaluation, limitations, human oversight, monitoring, incident response, and more).
8. Where AI is actually used
Everything above runs without any AI model. The features that do send your input to a third-party LLM provider are the Governance Copilot (chat answers), the Governance Agent / Agent Loop (drafted remediation steps), and the AI-written blog — each one is labeled with a "Third-party AI processing" badge in the Workbench (see our subprocessor list). If no provider is configured on a given deployment, these features fall back to a deterministic, server-only answer rather than silently failing.
9. Limitations & disclaimers
- These are educational risk-triage aids, not legal, compliance, or professional advice — see Terms §3.
- Scores and classifications are only as accurate as the answers and data you provide; we do not independently verify your inputs.
- The Fairness Scanner and Drift tools analyze whatever sample you load — a small or unrepresentative sample will understate real-world disparities or drift.
- Regulatory mappings reflect our curated reading of public sources as of each entry's last-verified date; they are not a substitute for legal review, and proposed legislation is explicitly labeled "Proposed" rather than treated as binding — see the status badges on each matched regulation in the System Intake and Compliance Dossier panels of the Workbench.
- Trust Passports and readiness scores reflect self-attested inputs unless a specific engagement says otherwise — see Terms §8.
10. Contact
Questions about how a specific number was calculated: info@autogovern.io.