There is no partial credit in de-identification

Over the past two months my team benchmarked the new general-purpose privacy tools on clinical text: OpenAI’s Privacy Filter at 0.55 PHI F1, Microsoft Presidio at 0.60 to 0.85 F1 in independent peer-reviewed studies, Databricks’ `ai_mask()` at 0.71, and the frontier LLM APIs at 0.86 to 0.91, against 0.96 for a healthcare-specific pipeline on the same corpus, which also holds 0.98 recall on a second 381,959-token clinical corpus and 0.98 micro F1 on the official 2014 i2b2 test set. The per-entity tables are on the John Snow Labs blog. This post is about how to read them, because de-identification is one of the few tasks in machine learning where the reading is pass or fail. A summarizer at 0.85 saves a team real effort. A de-identification pass at 0.89 produces a corpus that no privacy officer can release, which is the same thing a pass at 0.55 produces.

The legal standard: no PHI or very low risk of it

The regulation does not ask for a score. HIPAA offers two paths to de-identified status: Safe Harbor, which requires that all 18 identifier categories be removed, and Expert Determination, in which a qualified statistician certifies that the risk of re-identification is very small. Under GDPR, data that has been pseudonymized rather than anonymized remains personal data inside the full regulatory perimeter. Both frameworks are binary. No tier of either recognizes 80% removal and grants 80% of the permissions, and there is no version of “mostly de-identified” that a downstream researcher, partner, or model-training pipeline is allowed to touch.

Neither framework names an accuracy number, because neither was written about software. The field calibrated instead against the only prior benchmark that existed, careful manual de-identification by domain experts, and Yogarajan, Pfahringer, and Mayo’s 2020 review identifies 95% F1 on the 2014 i2b2 corpus as the level widely treated as equivalent to it.

95% is hard to reach with people

That bar sits above what individual experts achieve. Neamatullah et al. measured 14 clinicians de-identifying nursing notes by hand: recall ran from 0.63 to 0.94 by clinician, a single annotator averaged 0.81, two-annotator consensus reached 0.94, and even a consensus of three experts failed to remove all PHI. Building the i2b2 gold standard, Stubbs and Uzuner found individual annotators averaged 0.93 token F1 against the consensus they collectively produced, and the corpus reached gold quality only through double annotation of every record, arbitration, and rounds of sanity checking and proof reading. A single reviewer is not a safety net: one identifier in five gets past them.

What a recall number means at a million notes

Recall carries the regulatory risk and its implication multiplies across a corpus. A clinical note carries roughly 31 PHI mentions.

* Chunk-level detection, counting partial overlaps as found; on exact matches only, ai_mask() sits at 0.51.

The last column is recall^31, the chance a note comes out fully clean if each mention were independent, which overstates the damage since a repeated name is one detection problem rather than eight (read it as a worst case). The direction survives any correction. Below about 0.90, almost no document in a corpus is clean, and no downstream process recovers it. Above 0.95, the residue becomes small enough for adversarial testing and statistical risk analysis to characterize, which is what Expert Determination requires.

John Snow Labs’ pipeline sits above that line on every corpus tested: 0.98 recall in the table above, 0.98 micro F1 on the official i2b2 test set, and over 99% detection across 2 billion notes at Providence. Between 0.72 and 0.98 there is no plateau at which the data becomes shareable.

0.89 fails the same way 0.55 fails

The frontier models did improve. GPT-4o scored 0.79 PHI F1 in our Text2Story 2025 evaluation; GPT-5.5 scores 0.89 on the same corpus today. Read as a trend line, that number invites a plan: wait a version or two and let the general models solve it. Read against the bar, it says something different. 0.79 was below the threshold and 0.89 is below the threshold, and both produce a corpus that holds no de-identified status and cannot leave the security perimeter. In this task, improvement below the bar has a regulatory value of zero.

The residual errors also sit in the worst possible place. The aggregate hides it and the per-entity tables show it: Claude Opus 4.8 scores 0.49 on contact identifiers and 0.68 on age, and Gemini 3.1 Pro scores 0.60 on age, against 0.95 and 0.97 for the healthcare pipeline. Contact and age are enumerated Safe Harbor categories, and age carries its own aggregation rule above 89. These identifiers are defined by their role in a document rather than their surface form: phone and fax numbers live in headers, footers, and referral blocks, and record numbers appear as MRN, MR#, and a dozen local conventions. That’s a structural mismatch with general training distributions – and why the last points of the gap are the hardest ones, concentrated on the categories the regulation defines.

In addition, a model that cleared the number would still not clear the bar, because