Matching benchmark & methodology
How good is the matching? We validated the engine against an independent 1,736-name benchmark — names screened through a well-known reference screener (OpenSanctions) — and measured recall and precision against its verdicts. The numbers below are our own measurements, reproducible with the playground or the API. Everything was run against production.
Headline (Recall-first default)
Measured by actually screening every name at threshold 0.70. Two recall figures matter: overall recall (above, vs the reference's full database) and recall on entities already in our data (~95.8%, essentially flat across thresholds). The gap between them is data coverage — the reference screens a larger commercial database — closed with a bring-your-own feed, not by tuning. See methodology.
The recall-vs-precision curve
Measured by actually screening all 1,707 names at each threshold (not extrapolated). Raising the threshold trades overall recall for precision: precision is already ~100% by 0.80, while recall keeps falling. The default sits at 0.70 — the recall-maximizing end. The dot at 0.85 marks the old default, for reference.
The three presets
Every tenant starts on Recall-first. Switch presets, save your own named profiles, or reset to ours from the playground or the risk-profile API. Presets differ only in the fuzzy threshold — the lever the sweep actually varied — so the figures map directly to your choice.
| Preset | Threshold | Overall recall | Genuine precision | Evasion catches | Use it when |
|---|---|---|---|---|---|
| Recall-first Default | 0.70 | 95.1% | 96.8% | 26 | You can't afford to miss anyone — reviewers clear the few near-name collisions. |
| Balanced | 0.80 | 92.8% | 100% | 20 | Strong recall with zero unexplained false positives. |
| Precision-first | 0.90 | 86.3% | 100% | 18 | High-volume, lower-risk screening where review time is the constraint. |
Evasion catches — beating the reference
The benchmark includes deliberately fuzzed names — leetspeak and injected digits that defeat the reference screener's phonetic matching. Our normalizer strips them and still matches the real listed entity. These are wins, not false positives: each is a real sanctioned or politically-exposed person we flagged at high confidence. A sample (names from public-domain sanctions lists & CC0 Wikidata):
| Fuzzed input | Real listed entity we matched | List | Confidence | |
|---|---|---|---|---|
| Sin Kyu-N4M | → | Sin Kyu-Nam | EU | 79% |
| Mah4R… | → | Maher Hashem Katranji | OFAC | 97% |
| Ydlena Gennadyevna Zlenko | → | Yelena Gennadyevna Zlenko | UK | 77% |
| Wagn… | → | Wang Zhiqing | OFAC | 97% |
| Jogre… | → | Jorge García Carneiro | Wikidata (PEP) | 98% |
26 such catches at the default setting. Fuzzed inputs are our own synthetic test strings; the matched names are public government / CC0 data.
Methodology
- Benchmark: 1,736 names screened through OpenSanctions; its verdicts (found / not-found, with topics and scores) are the reference labels.
- What we measure: we screen the same names through
POST /v1/screenings/checkand compare. A disagreement is bucketed as a matching gap (the entity is in our data but we passed it — a real tuning target) or a coverage gap (the entity isn't in our free sources at all — answered by a BYO feed, not tuning). - Overall recall = hits ÷ all golden-flagged names. The figure the threshold moves. Recall in-data = hits ÷ (hits + matching gaps) — essentially flat (~95.8%) across thresholds, because the in-data misses are classification/coverage, not score cutoffs.
- Genuine precision: a flag on a benchmark "clean" name only counts against us if it isn't backed by a real listed match. Whether a flag is a correct evasion catch or a genuine collision is decided by name similarity between the input and the matched entity — so a legitimate catch scoring 0.70–0.85 isn't mislabeled a false positive.
- Measured, not extrapolated: each threshold's numbers come from actually screening all 1,707 names at that threshold — not from re-scoring one run. The residual collisions at 0.70 are near-identical surnames in the large CC0 PEP set, which a reviewer clears in seconds.
- Reproduce it: screen any name in the playground, or run the published harness against the API.