Generative AI Hallucinates Biological Data, Creating Fake Research

Generative AI systems can confidently invent biological discoveries that do not exist, presenting them as fact to researchers and students.

What Happened

Researchers recently tested a large language model to see if it could help invent new scientific hypotheses. The model was asked to search scientific databases and generate new protein structures and chemical reactions. The model produced results that looked plausible. It used the correct scientific language and cited references. However, the discoveries were entirely fake. The model had generated these results by predicting the next most likely word in a sequence, not by retrieving verified facts. It invented data that did not exist in reality.

The Governance Failure

The failure here is a lack of provenance and grounding. The system took unverified data and treated it as a primary source. This mirrors the Air Canada case where a support chatbot invented a bereavement-refund policy. The airline was held liable for what its AI told a customer. In both cases, the organization assumed liability for outputs that the AI produced without a human verifying the truth. The risk is that a researcher might act on this false information, wasting time and resources or leading to failed experiments.

Risk Pattern

This exemplifies the risk of blind trust in generative outputs. It is not enough to simply use an AI tool. The governance must cover the verification of the output. We cannot treat the AI as a peer reviewer or a database. The pattern shows that without strict controls, AI will confidently present errors as facts. This breaks the chain of trust that organizations rely on to make decisions.

Key Controls

We need to ensure all AI-generated content is clearly tagged. The system must be forced to cite sources or link back to the original data. We need a human sign-off process before anything is published or used for critical decisions. This applies to both internal research and public-facing content. We must treat AI outputs as suggestions to be verified, not as final answers.

What to do

• Tag all AI-generated content clearly as AI-generated
• Require citations or source links for every claim
• Implement a human sign-off process for critical outputs
• Audit your RAG systems to ensure they are not regenerating hallucinations
• Treat AI outputs as suggestions to be verified, not final answers

What to do

• Tag all AI-generated content clearly as AI-generated
• Require citations or source links for every claim
• Implement a human sign-off process for critical outputs
• Audit your RAG systems to ensure they are not regenerating hallucinations

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.