One detector, one set of images, three published numbers: 1.000, about 0.70, and 0.243. Nothing changed but the preparation of the files. A procurement process that records the first number and discards the condition has not recorded a result, it has recorded half of one.
Three scores, one detector
In our benchmark of fourteen open-source detectors, DMimageDetection scored an AUC of 1.000. AUC runs from 0 to 1: it is the chance the model ranks a randomly chosen synthetic image above a randomly chosen real one, so 0.5 is a coin flip and 1.000 is perfect separation. That is the number a datasheet would carry.
Passed through our four emulated platform pipelines, the same detector scores around 0.70. Still useful, clearly degraded, and a different procurement conversation from a perfect score.
Passed through a single shared encoder, so that real and synthetic images were saved identically, it scores 0.243. Below 0.5 a classifier is not guessing, it is systematically predicting the wrong class. The ClipBased fusion checkpoint follows the same path, from 1.000 to 0.340.
Same model. Same images. 1.000, 0.70, and 0.243. The only variable across the three is how the bytes were written.
The control that makes this attributable is the dull part and the load-bearing one. Running the original files through the new harness without re-encoding reproduced the first-round scores to four decimal places, which eliminates the harness as an explanation and leaves the file preparation as the only remaining variable.
The wrong suspect
The intuitive story is that platform compression breaks detectors. Everyone believes it, we half believed it, and our own results say it is not what happened here.
Our platform pipelines start from the original image bytes, so they carry forward the difference in how real and synthetic images had been saved. The detectors still read that difference, and they scored around 0.70. It was the condition that removed the difference, by putting both classes through one encoder, that took them below random.
Read carefully, that is worse news than the intuitive version, not better. A detector that survives platform processing but collapses under format parity was never reading synthesis. It was reading the file. Platform compression left its shortcut intact; only the control condition took it away.
It also marks the boundary of what we measured. A live platform re-encodes everything it ingests through one pipeline, which is closer to the condition that caused the collapse than to the one we emulated. Whether that holds end to end in production is untested, and we are not going to claim it from these results. The mechanism is demonstrated. The production number is not.
What the survivors show
Six of the eight detectors carried into the second phase did not collapse. They lost 0.047 AUC on average under format parity, worst case 0.119, and held steady across the platform pipelines with a standard deviation of 0.011. Whatever they were reading, it was not the encoder.
That spread is the useful part of the study. Two detectors with identical headline numbers can be doing entirely different things, and no amount of staring at the headline will tell you which. The single-axis sweep makes the same point from another angle: a 0.5 resize cost most detectors 10 to 20 AUC points, while JPEG compression broke none of them above quality factor 70. The perturbation everyone worries about was not the dangerous one.
What we control for
This result is the reason our own protocol looks the way it does. Every evaluation we run prepares real and synthetic samples through one identical path, so no format shortcut is available to be scored. Every number we report carries the condition it was measured under, and where there are several conditions we publish several numbers rather than the flattering one. Results come broken out by group, worst case named, not pooled into an average. And a control run establishes that the harness itself did not move the result before any finding is attributed to the data.
None of that is exotic. It is what this study forced on us after a perfect score turned out to be measuring the file.
Underneath all four is the rule this work keeps returning to. The question is never whether a detector scores well, it is whether it reads a property the attacker cannot supply, and a number reported without its condition cannot tell you which it did.
This is the same argument we make about which statistic to optimize for identity verification, one layer down. That piece is about choosing the right metric. This one is about the fact that any metric, correctly chosen, still means nothing until the condition is attached to it. The full results and methodology are in the benchmark paper.
Margen does not sell a detector. We measure them under stated conditions and publish the conditions alongside the number, because one without the other is not a result.
Related reading
- Methodology notesAUC is the wrong number for identity verification.Thirteen published synthetic-ID detectors, measured on 6,948 passport-style images. Nine change rank the moment you measure what they actually catch rather than how they rank, and that gap is visible before a document has been through a single scanner.
- Fraud storiesA cloned voice that said the password, and the bank let it in.Reporters cloned their own voices and passed bank voiceprint authentication. Researchers defeated it with up to 99 percent success in six tries. The industry now concedes the control is beaten.
- ExplainersMeta's Muse watermark, and what it does not do for deepfake detection.Meta's Content Seal watermark labels the content Meta itself generates. That is useful, and it is not the same as detecting the deepfakes an attacker actually makes.