Workday's Discovery Wins Show the Real Problem: You Can't Audit a Black Box

The case: Workday wins on discovery, plaintiffs lose on evidence

A federal court recently ruled in a proposed class action against Workday, the HR software giant. The plaintiffs claim its AI screening tools discriminate on the basis of race, age, and disability. Workday asked the court to limit discovery, and the court sided with Workday. The plaintiffs will not get the full access they wanted to Workday's internal model details, training data, and testing records.

The case is not over. But the ruling sets a pattern: courts are wary of forcing vendors to hand over proprietary AI systems in bias lawsuits. That means plaintiffs have to prove discrimination without seeing the model. They have to rely on outcomes, not internals.

That is a huge deal. And it is not just a legal problem for plaintiffs. It is a governance problem for every company that uses a vendor's AI to make decisions about people.

What actually happened, mechanically

Workday sells software that helps employers screen resumes, rank candidates, and manage performance. The plaintiffs say that software produces biased results, and they wanted discovery into how the models work. Workday pushed back, arguing that its source code, training data, and internal testing are trade secrets. The court agreed, at least for now, and limited what the plaintiffs can demand.

This is not a case where a model was shown to be biased. It is a case where the plaintiffs cannot get the evidence to show bias. The mechanism is the same in every AI bias claim: you need to see the inputs, the model logic, and the outputs to prove disparate impact. If you cannot see them, you are left with statistical patterns and hope.

The court's decision does not say Workday's tools are fair. It says the plaintiffs have to fight with one hand tied behind their backs. That is a discovery problem, but it is also a transparency problem.

The failure mode: you cannot audit what you cannot see

This case is a textbook example of the opacity failure mode in AI risk management. When a vendor sells you an AI system, you do not get the model. You get an API and a contract. If the model produces biased outcomes, you cannot see why. You cannot audit it. You cannot prove it. And if a plaintiff tries to prove it, the vendor will fight to keep the model hidden.

The precedents are clear. Northpointe's criminal risk score in 2016 showed large racial disparities in false positives, but the company never fully disclosed its logic. Goldman Sachs faced a regulator probe in 2019 after reports that women got far lower credit limits than men, but the models were opaque. iTutorGroup settled with the EEOC in 2023 after its recruiting software automatically rejected older applicants, but the internal threshold was only revealed through investigation.

In every case, the harm was real, but the evidence was buried. The pattern is consistent: bias hides in opaque models, and discovery is the only way to surface it. When courts limit discovery, they are not just limiting litigation. They are limiting accountability.

What governance obligations apply

If you use a vendor's AI to make employment decisions, you are on the hook. The EU AI Act's high-risk rules, which apply to employment and recruiting systems from December 2027, require you to run a fundamental rights impact assessment, ensure human oversight, and maintain technical documentation. But the Act also requires you to test for bias and to be able to explain decisions. If your vendor hides the model, you cannot meet those obligations.

In the US, the EEOC has made clear that AI screening tools must comply with anti-discrimination laws, including the Age Discrimination in Employment Act and the Americans with Disabilities Act. The 4/5ths rule, a disparate impact test, applies to employment decisions. If your vendor's tool produces a selection rate for a protected group that is less than 80% of the rate for the majority group, you have a problem. But you cannot run that test if you do not have the data.

The key risk indicators are simple: the 4/5ths ratio per protected group, the equal opportunity gap between groups, and the percentage of adverse decisions overturned on human review. If you cannot measure those, you are flying blind.

What to do

1. Put audit rights in your vendor contracts. Before you sign, require access to model documentation, testing results, and the ability to run your own fairness tests on your own data. If the vendor refuses, walk away.

2. Run your own outcome-based bias tests. You do not need the source code. You need the decisions. Run a 4/5ths analysis on your own hiring data, by race, age, and sex. If you see disparities, investigate before a plaintiff does.

3. Keep a human in the loop. The EU AI Act requires human oversight for high-risk systems. Make sure a human reviews every adverse decision that could be challenged. Track the overturn rate. If it is high, your model is probably wrong.

4. Document everything. If you are ever sued, your defense is your audit trail. Show that you tested, you found issues, and you fixed them. That is the difference between a compliance failure and a governance success.

5. Push for industry standards on transparency. This case will not be the last. Vendors will keep hiding models until buyers demand otherwise. Your procurement decisions are the strongest lever you have.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.