November 20, 2026
EngineeringHow we stress-test Helios against real attackers
By Basab Jha, Ujjwal Puri and Gagan Rai, Members of Technical Staff
Identity verification is the front door to a bank, a wallet or an insurer. If someone gets through it, every control behind it inherits the mistake. That makes a clean test set a poor measure of readiness: real attackers do not submit clean inputs, and they do not stop at the first thing that works.
Helios is our identity verification infrastructure, and we evaluate it the way an attacker would meet it. Over the past year that meant adversarial evaluation at three layers — the models, the system built around them, and the way it is deployed and operated — alongside an independent penetration test of the full product surface.
Across 14 attack classes and 46,300 attack instances, the success rate of our most capable attacker tier fell from 11.8% to 0.9% over four evaluation cycles, while false rejections for genuine users stayed within 0.4 percentage points of baseline. The independent assessment returned 25 findings, three rated high and none critical, all of which were remediated and confirmed on retest.
This post is about how we measure, not about what Helios is made of:
- How we build attack sets, define attacker tiers and report results without flattering ourselves.
- How we run an independent assessment, and how its findings become permanent regression coverage.
Any single measure here is limited. The point is that together they compound into something we can defend with data.
Three layers, one library
Verification fails in more than one place. A model can be fooled. A system can be bypassed even when every model behaves correctly. A deployment can be misconfigured in ways no model test will ever catch. These are different failures, so we measure them separately — and then feed all three into one shared attack library, so a lesson learned in one layer becomes a test in another.
Two kinds of attack, two different metrics
Attacks on a verification system fall into two families. They fail in different ways, so no single number can describe both.
Presentation attacks happen in front of the camera: a printed photo, a replayed video on a screen, a mask, a document recaptured off a monitor. We report these in the vocabulary of ISO/IEC 30107-3 — APCER, the share of attacks wrongly accepted as genuine, and BPCER, the share of genuine users wrongly rejected. The two are always reported together at a fixed operating point, because either one alone is trivial to game. A system that rejects everyone has a perfect APCER.
Injection attacks happen behind the camera. The attacker supplies synthetic or manipulated input through a path that bypasses the sensor entirely, so there is no physical artifact to detect. Here the meaningful quantity is attack success rate, measured per technique and per environment, in the shape the emerging injection-attack standards describe. A model evaluated only on presentation attacks tells you very little about this family.
An attack set that can embarrass us
An evaluation is only as good as the attacks it contains, so most of our effort goes there rather than into the metric.
We organize attacks along four axes: the attack class, the capture environment (device quality, lighting, network conditions), the document type and script involved, and the attacker’s level of effort. The current set covers 14 attack classes and 46,300 captured instances, drawn from a device pool that runs from low-end handsets to recent flagships, and balanced against a genuine-session corpus several times its size.
Effort is split into tiers. The low tier is what anyone can do in minutes with a phone and a printer. The middle tier uses readily available tooling and some preparation. The high tier assumes a motivated attacker with purpose-built synthetic media and custom tooling. We report success rates per tier, because an average hides exactly the tail an institution cares about.
Two practices matter most. First, we hold out entire attack families: models are scored on techniques they were never trained against, because a system that only recognizes yesterday’s attacks is a lookup table. In the latest cycle, 3 of the 14 classes were held out entirely. Second, we test the genuine side as hard as the attack side — low-end cameras, poor light, older users, worn documents. In onboarding, a false rejection is a lost customer and an opened support ticket, and security bought with unusable friction is not a gain.
Small denominators need care. A zero is not a zero rate: with n attempts and no observed successes, the 95% upper bound on the true rate is roughly 3/n — the rule of three — so zero successes in 1,200 attempts bounds the true rate below 0.25%. We publish that bound next to the observation, and Wilson or Clopper-Pearson intervals everywhere else.
Documents fail in their own way
Documents break differently from faces, and they run through both families above: a recaptured document is a presentation attack, an injected synthetic one is an injection attack. We evaluate three techniques — recapture, tampering (altered fields, substituted portraits) and fully synthetic documents.
Tampering and synthesis give us controlled sets with known ground truth. Recapture and real-world abuse give us sessions that human reviewers have examined, double-annotated and adjudicated, because label quality bounds everything downstream.
Results are broken down by document type, script and handwriting presence rather than pooled. A detector that is strong on one document family and weak on another is a finding worth publishing, and pooling would bury it.
Bringing in outside eyes
Internal red-teaming has a structural limit: the people who built the system also built the assumptions it is tested against. So we commissioned an independent assessment from a firm with no hand in building Helios. It ran for three weeks against the production release current at the time, in an environment that mirrored production configuration, using synthetic identities and no customer data.
Scope covered the mobile SDKs, the web SDK, the API and callback surface, the back-office console, and deployment configuration. Test cases were mapped to OWASP ASVS, MASVS and WSTG. Findings were rated both by severity and by whether they were actually reachable in our deployment — an issue that looks alarming in isolation can be unreachable in practice, and the reverse happens too.
Every finding was remediated and re-tested before publication. The outcome:
| Severity | Found | Closed | Confirmed on retest |
|---|---|---|---|
| Critical | 0 | 0 | 0 |
| High | 3 | 3 | 3 |
| Medium | 8 | 8 | 8 |
| Low / informational | 14 | 14 | 14 |
The finding that mattered most
The most valuable finding was not a classic vulnerability. It was a logic issue: a sequence of requests that were each valid on their own could leave a session in a state we did not handle as intended. Nothing was bypassed outright — every model scored correctly — which is exactly why it had survived our own review. We fixed the behavior, wrote the sequence into the attack library as a new class, and that class immediately surfaced two related variants our evaluation set had never covered.
That is the loop in miniature. Every closed finding becomes a regression test; every technique a tester used joins the library the model evaluations draw from. The penetration test sharpens the evaluation set, and the evaluation set keeps the penetration test’s lessons from decaying.
Every number has an expiry date
An assessment describes one version of a system at one point in time. An attack set describes the attacks someone thought of. Synthetic media improves quickly, so we treat every metric here as perishable rather than as a property of the product.
So evaluations re-run every quarter and on material releases. Escalated production sessions are reviewed for patterns that were never in the set. Confirmed attacks go back into the library. Publishing the numbers is not a claim that the problem is closed; it is how we keep ourselves from quietly drifting away from them.
Assurance is a loop, not a score
The assurance we can offer is the product of evaluation, independent testing and operations compounding on each other — not of any single test. As attacks evolve, we expect the loop between them to tighten. We will keep publishing what we measure, including when the numbers move the wrong way.
Security researchers who believe they have found an issue in Helios can report it through the process described on our security page. We read every report.
- Engineering
- 2026

