NewThe detectors that scored perfect collapsed the hardest under attack.
Back to publications
Explainer

What is an operating point?

An operating point is the threshold that turns a detector's score into a decision. The default of 0.5 is arbitrary, because a detector's output is not a calibrated probability, so the threshold has to be set from the error you cannot tolerate.

Explainer · September 2026 · 5 minute read

The definition

A detector does not return an answer. It returns a number, usually between 0 and 1, which is higher when the model finds the image more suspicious. Somewhere a line has to be drawn: above this, flag it; below, let it through.

The operating point is that line. It is the threshold at which a score becomes a decision, and it is a choice made by whoever deploys the model, not a property of the model.

Every performance figure that describes behaviour rather than ranking is measured at an operating point. False positive rate, false negative rate, precision, recall, accuracy: all of them move when the threshold moves, on the same model and the same images.

Why 0.5 means nothing

The common default is 0.5, on the reasoning that a score above half means the model thinks it is more likely fake than real. That reasoning assumes the output is a calibrated probability. It usually is not.

A detector is trained to separate two classes, not to report honest probabilities, and the raw output is whatever the final layer happens to emit. A model can put most of its genuine images between 0.6 and 0.9 and still rank every one of them correctly below its fakes. Its 0.5 and another model's 0.5 are not the same quantity, and neither corresponds to a fifty-fifty belief about anything.

The practical consequence is that a threshold carried over from one model to another is a guess. This is also why scores from several detectors cannot be pooled into a single curve: the between-model offset dominates, and the result measures calibration mismatch rather than detection.

Setting one on purpose

A threshold trades two errors against each other, and moving it cannot reduce both. Push it down and more fakes get caught while more genuine users get rejected. Push it up and the reverse.

So the threshold is set from whichever error the system cannot afford. In identity verification the two are not close. A false negative admits a synthetic identity into an account, and a false positive asks a real customer to retake a photo. One is a loss and the other is friction.

The usual method is to fix the error you can tolerate and read off the other. Choose an acceptable false-positive rate on genuine inputs, find the threshold that produces it, then measure recall, which is the fraction of fakes caught at that setting. That second number is the one that describes the control.

The rule we use

In our passport and ID benchmark, each detector's threshold is the 95th percentile of its own scores over the clean genuine images. By construction that is a 5 percent false-alarm rate on clean inputs, and because it is computed per detector it lands each model at a comparable level of friction rather than a comparable number.

That last point is the reason for the rule. Thirteen detectors with thirteen different score distributions cannot be compared at a shared numeric threshold, because the same number means something different in each. Fixing the false-alarm rate instead puts them on common ground: every model is asked how many fakes it catches while inconveniencing the same share of genuine users.

Doing that changed the leaderboard. Scored on what they catch at a fixed threshold rather than on how they rank, nine of the thirteen detectors changed position. A ranking statistic and a decision statistic are answering different questions, and the gap between them is visible on clean files, before any document has been through a scanner.

Why it then stays fixed

Once set on clean inputs, the threshold is held constant across every other condition tested: compressed, printed, photocopied, re-encoded.

Re-tuning per condition would flatter every model, because the threshold would silently absorb whatever the condition did to the score distribution. The result would describe how well the threshold was re-fitted rather than how the detector held up. Holding it fixed makes the degradation visible, which is the thing being measured.

It also mirrors deployment. A production system does not recalibrate because a user photographed their passport under a desk lamp. The threshold was chosen once, and everything arriving afterwards is judged against it. Whether a detector was reading a durable property or something incidental to the test set tends to surface the moment conditions move, and a fixed threshold is what makes that surfacing legible.

Related reading