Why Deepfake Detection Alone Does Not Stop Fraud

Group-IB's analysis of deepfake fraud makes a blunt point: detection tools alone do not stop it, because the fraud does not depend on the fake being perfect. It depends on one person believing one convincing call.

The mechanism, not just the outcome

A deepfake fraud usually runs in three steps. First, the attacker gathers material: a few seconds of someone's voice from a podcast or a video call, a photo, a job title. Second, they impersonate that person on a single channel, often a voice call or a video meeting, and create urgency. A payment needs to go out now, a supplier's bank details have changed, the boss is travelling and cannot be reached. Third, the target authorises the action because the request came from someone who sounded and looked right.

The fake does not have to fool a detection model. It has to fool a human under time pressure, and it only has to work once. That is why Group-IB argues detection alone fails. Detection is a filter on the input. The fraud is a failure of the process that lets one channel authorise money or access.

The risk management failure it exposes

This is a single-channel authorisation problem. If a payment, a password reset, or a change of bank details can be approved on the strength of one phone call or one video meeting, then the control is a person's ear, and a person's ear can be fooled.

The pattern is familiar from other AI failures. Air Canada's support chatbot invented a bereavement policy, and a tribunal held the airline responsible for what its AI told a customer. The lesson there was that the organisation owns what its systems say. The lesson here is similar: the organisation owns the loss when its process lets a convincing voice move money.

The controls that actually apply

• Out-of-band verification. Any change of payment details or sensitive action needs confirmation through a second channel the caller does not control, such as a callback to a known number.
• Dual authorisation for high-value actions, with a cooling-off period for bank detail changes.
• A shared code word or challenge question agreed in advance with regular counterparties.
• Staff training that names the pattern: urgency, secrecy, a change of account, a request that bypasses the normal process.
• A fast way to warn staff after a confirmed impersonation attempt, because the same actor often tries several people.

On the regulatory side, the EU AI Act requires providers of certain AI systems to mark synthetic content in a machine-readable way, and that transparency rule has been in force since 2 August 2026, with a retrofit deadline of 2 December 2026 for systems already on the market. That helps platforms label fakes, but it does not help the person on the call. Your process controls do that.

Measuring whether you are exposed

Three indicators tell you where you stand. First, how many sensitive actions can be authorised on a single channel. The target is zero. Second, how long it takes to warn staff after a confirmed impersonation attempt. Third, what share of synthetic media you publish carries machine-readable marking, which is both a legal obligation and a way to protect your own likeness from being cloned.

A control is only as good as the threat it is sized against. ThreatClaw, at www.threatclaw.ai, tracks the threat side of these systems with live intelligence feeds and deployable detection rules. For the filtering question, its article at https://www.threatclaw.ai/blog/the-invisible-gate-filtering-what-your-ai-says-before-it-does-damage is a useful read.

What to do

• Map every action that can be authorised on a single channel: payments, bank detail changes, password resets, access grants. Set a target of zero and close the highest-value ones first.
• Require out-of-band confirmation for any change of payment details, using a number you already hold, not one the caller gives you.
• Agree a code word with regular counterparties and tell staff to use it when a request feels urgent or unusual.
• Write a one-page warning template now, so you can send it to staff within an hour of a confirmed impersonation attempt.
• Check whether the synthetic media you publish carries machine-readable marking, and fix the ones that do not before the 2 December 2026 deadline.

A governance or risk program, or a tool like autogovern.io, can help you keep these controls and indicators in one place.

More from our platforms

These sister platforms cover the parts of this problem that sit outside governance.

• Argus (argus.threatclaw.ai) records every trace an AI application produces and scans it for prompt injection, jailbreaks and data leaks, including the attacks hidden inside retrieved documents and tool results rather than in what the user typed. Governance decides what an AI agent is allowed to do. Argus shows what it actually did.
• ThreatClaw (www.threatclaw.ai) tracks the threat side of the same systems: 22 live intelligence feeds, exploitation predicted before it is officially confirmed, threat actor profiles, and detection rules you can deploy straight away. A control is only as good as the threat it is sized against.
• Xodexa (xodexa.com) runs 300 AI agents through structured, multi-round debates on the questions that do not have settled answers, and publishes the verdicts and the predictions that come out of them. Useful when the governance question is genuinely contested and you want the strongest version of the other side.

Related reading:

• The Invisible Gate: Filtering What Your AI Says Before It Does Damage on ThreatClaw