How an Algorithm’s Past Bias Still Haunts Us Today

What happened

The case study from City St George’s, University of London, shows how an AI system used historical data that included past discriminatory practices. The algorithm learned these biases, leading to unfair outcomes for certain groups today. This mirrors past incidents like Amazon’s 2018 resume screener, which penalized female candidates, and Northpointe’s 2016 criminal risk score, which disproportionately flagged minorities.

The governance failure

The core issue is a failure to audit training data for historical bias. When an AI learns from past decisions that were themselves biased, it replicates those biases. This is a classic example of ‘garbage in, garbage out’—but with serious ethical consequences. The system was likely deployed without checking whether its training data reflected fair, current standards.

Controls and obligations

Before deploying any AI, you must audit training data for bias. Check if certain groups are underrepresented or if historical discrimination is embedded in the data. The EU AI Act (Article 10) requires ‘data quality and suitability’ checks, meaning data should not contain systemic bias. For high-risk systems (Annex III), you must also test for fairness across groups (Article 14) and ensure human oversight in critical decisions (Annex III §8).

In the US, the Equal Employment Opportunity Commission (EEOC) guidelines and the Equal Credit Opportunity Act (ECOA) both prohibit using data or models that discriminate. A ‘4/5ths rule’ test—comparing decision rates between groups—can flag disparate impact early.

Key risk indicators

Track metrics like the 4/5ths disparate-impact ratio, the true-positive rate gap between groups, and the percentage of adverse decisions overturned on human review. These show whether your AI is treating groups fairly or repeating historical mistakes.

What to do

1. Audit your training data for historical bias before deployment. Remove or adjust data that reflects past discrimination.
2. Conduct fairness audits across all protected groups. Use tools to detect and mitigate bias.
3. Implement human oversight for high-stakes decisions. Ensure a person reviews outcomes that significantly affect individuals.
4. Stay compliant with the EU AI Act’s fairness requirements and US anti-discrimination laws.
5. Monitor risk indicators regularly. Act if any metric shows unfair outcomes.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.