NewThe detectors that scored perfect collapsed the hardest under attack.
Back to publications
Explainer

What is AUC?

AUC is the probability that a detector scores a randomly chosen fake above a randomly chosen real image. It runs 0 to 1, where 0.5 is a coin flip and anything below it means the model is systematically wrong rather than merely weak.

Explainer · September 2026 · 5 minute read

The definition

AUC stands for area under the curve, and the curve is the receiver operating characteristic, usually shortened to ROC. Most people never need the curve itself, because the number has a plain-English meaning.

AUC is the probability that the detector gives a higher score to a randomly chosen fake image than to a randomly chosen real one.

That is the whole idea. Take one fake and one real at random, ask the model to score both, and see whether it puts the fake higher. Do that for every possible pairing and the fraction it gets the right way round is the AUC.

Notice what the definition does not involve: any threshold. AUC asks only whether the model ranks fakes above reals, never whether it would actually flag any particular image. That property is why AUC is popular, because it summarises a model without committing to a decision rule, and it is also the source of most of the confusion around it.

Reading the scale

AUC runs from 0 to 1, and three points on that range carry all the meaning.

  • 1.000 is perfect separation. Every fake scored above every real, with no overlap.
  • 0.5 is a coin flip. The model ranks a fake above a real exactly half the time, which is what random guessing produces.
  • 0 is perfectly backwards. Every real scored above every fake.

A useful instinct: the distance from 0.5 is the signal, not the distance from 0. A detector at 0.75 is halfway between guessing and perfect, not three quarters of the way.

What it means to score below 0.5

This is the part that surprises people, and it is worth being precise about, because a score under 0.5 is not the same as a bad score.

A model at 0.5 has no information. A model below 0.5 has information and is using it backwards: it is reliably ranking real images above fakes. Something in the data is separating the two classes, and the model has learned to point at it the wrong way round.

We have a measured example. In our benchmark of fourteen open-source detectors, one scored an AUC of exactly 1.000, which is as good as the metric goes. Re-saving every image through a single shared encoder, so real and synthetic files were written identically, dropped the same model on the same images to 0.243. A second went from 1.000 to 0.340.

Nothing about the models changed. What changed was that the two classes had arrived in slightly different file formats, and the detectors had been separating images by that signature rather than by anything in the picture. Remove the signature and the leftover correlation runs backwards, which is what a score of 0.243 looks like from the outside.

What AUC does not tell you

Three things, each of which matters more in a deployment than the AUC itself.

It does not tell you where to set the threshold. AUC is computed across every threshold at once, so it says nothing about how the model behaves at the one you would actually run. That is a separate quantity, the operating point.

It does not tell you which group fails. A single pooled figure averages across everyone the system will meet. A model can post a respectable AUC overall while failing badly on one demographic group, and the average is what hides it.

It does not carry its condition. An AUC is a measurement of a model on a corpus prepared a particular way. The 1.000 and the 0.243 above are both real numbers for the same detector, and a figure quoted without the condition attached cannot tell you which kind you are looking at.

If you are buying a detector

AUC is a ranking statistic, and ranking is not what an identity- verification system does. It makes a decision at a fixed threshold, and the cost of its two possible errors is wildly unequal: letting a synthetic identity through is expensive, and asking a genuine customer to retake a selfie is not.

Because AUC weighs both directions equally across all thresholds, it can rank two detectors as near-equals while they behave completely differently at the point a real system runs. Scoring thirteen published detectors on what they actually catch at a fixed threshold rather than on how they rank, nine of the thirteen changed position. That piece works through which statistic fits the decision instead, and why the gap shows up before a document has been through a single scanner.

None of this makes AUC a bad metric. It makes it a summary of ranking ability, which is a genuinely useful thing to know and a different thing from whether a control holds.

Related reading