REDACT: Hidden Tokens on a third-party multilingual benchmark, losses included
- Someone else's benchmark. REDACT is a PII detection benchmark published by other researchers: 51 entity types, 25 languages, synthetic documents. Its authors score Presidio, GLiNER, the OpenAI Privacy Filter, GPT-4.1 and Claude Sonnet 4.6 on it. We ran Hidden Tokens on it and publish everything, including what is not good.
- Direct identifiers (names, emails, phone numbers, ID, account and badge numbers, addresses) fully disclosed, engine 2.85.1 alone: 99.2% touched by a code, 89.0% entirely hidden, 139 of 17,103 left untouched. With the Double-check (an optional second reader on the computer): 95.6% entirely hidden, 83 left untouched.
- Dates of birth left readable: 60.9% with connector 2.57.0, 41.1% with 2.85.1 alone, 5.1% with the Double-check.
- Exact decoding of a coded message: 372 of 1,000 with 2.57.0, 568 with 2.85.1 alone, 471 with the Double-check, which also costs a little over-coding and time (see below).
- Official typed score: 0.212 (0.223 with the Double-check), above Presidio (0.195), far below the language models (0.597 and 0.636). Hidden Tokens hides an entity without trying to name its exact type; this score penalises that, and we publish it anyway.
Summary, version by version
Each column is a run of the same harness on the same 1,000 records. Connector 2.57.0 is the one measured on 31 October 2026; 2.63.0 and 2.64.0 fixed what the previous run showed. 2.85.1 is the connector of 6 November 2026, measured alone and with the Double-check.
| Measure | Connector 2.57.0 | Connector 2.63.0 | Connector 2.64.0 | Connector 2.77.0 | Connector 2.77.0 + Double-check | Connector 2.85.1 | Connector 2.85.1 + Double-check |
|---|---|---|---|---|---|---|---|
| Fully disclosed direct identifiers touched by a code | 98.8% | 99.0% | 99.2% | 99.2% | 99.5% | 99.2% | 99.5% |
| … entirely hidden | 87.5% | 88.3% | 89.0% | 89.0% | 96.4% | 89.0% | 95.6% |
| … left untouched | 200 / 17,103 | 174 / 17,103 | 139 / 17,103 | 139 / 17,103 | 80 / 17,103 | 139 / 17,103 | 83 / 17,103 |
| Dates of birth left readable | 60.9% | 41.1% | 40.9% | 41.1% | 3.2% | 41.1% | 5.1% |
| Coded text that decodes back exactly (text route) | 372 / 1,000 | 542 / 1,000 | 549 / 1,000 | 561 / 1,000 | 549 / 1,000 | 568 / 1,000 | 471 / 1,000 |
| … (file route) | 765 / 1,000 | 748 / 1,000 | 751 / 1,000 | 776 / 1,000 | 716 / 1,000 | 552 / 1,000 | 465 / 1,000 |
| Typed score, official method (partial micro-F1) | 0.203 | 0.211 | 0.212 | 0.213 | 0.239 | 0.212 | 0.223 |
The Double-check: a second reader on the computer
The Double-check is an optional package (725 MB on Mac, 702 MB on Windows) that the Hidden Tokens app installs on request. A local model (GLiNER multi-PII, Apache 2.0) rereads the text before coding, on the computer, with no network. What it finds is added to the values to code and gets real codes, which decode. Once installed it is on by default; a folder can turn it off. Below: the same 1,000 records, the same connector 2.85.1, without it and with it. In this run it read 1,000 of 1,000 records (0 unavailable) and added 9,015 passages to the coding.
| Measure | Engine alone 2.85.1 | 2.85.1 + Double-check | Change |
|---|---|---|---|
| Direct identifiers left untouched | 139 | 83 | −56 |
| … entirely hidden | 89.0% | 95.6% | +6.6 pt |
| Dates of birth left readable | 41.1% | 5.1% | −36.0 pt |
| Ordinary text coded, French | 10.7% | 12.9% | +2.2 pt |
| … English | 23.6% | 27.2% | +3.6 pt |
| … average of the 25 languages | 56.0% | 58.9% | +2.9 pt |
| Exact decoding, text route | 568 / 1,000 | 471 / 1,000 | −97 |
| … file route | 552 / 1,000 | 465 / 1,000 | −87 |
| Official typed score (partial micro-F1) | 0.212 | 0.223 | +0.011 |
| Time per record, median (both routes) | 0.05 s | 0.89 s | — |
What it costs, and what is not right yet:
- A little more over-coding: +2.2 pt in French, +3.6 pt in English, +2.9 pt on average over the 25 languages. The model also codes a few words that are not personal data.
- Fewer texts that come back exactly: 97 files and 110 messages no longer decode back exactly (10 and 13 now do). 92 of those files and 103 of those messages differ only by capital letters: the "one spelling per code" rule (below), applied to more values. The meaning comes back, not always the case. The other 5 files and 7 messages differ in other ways; they are not analysed yet.
- "It only adds" holds: no value that the engine alone coded is left untouched with it, on either route. That was not the case in 2.77.0 (2 values: a Danish first name, an English customer reference): since 2.85.1 it is guaranteed by construction and checked word by word on every text, with the engine alone as the last resort.
- Time and memory: 0.05 s per record (median) without it, 0.89 s with it (95th percentile 4.3 s), both routes, on a Mac M5; about 1.1 to 1.3 GB of memory while reading (measured with the portable bench, on 100 records). Apple Silicon Macs and Windows x64 only, not Intel Macs. Windows was not measured on this split.
- On this split, it is blind. Its settings (threshold 0.5, "organisation" label left out) were chosen on 5 November 2026 on 2,600 records outside the split; these 1,000 records were never used to tune it.
What got worse, and what is still missing
- Some attributes written as ordinary words are now readable. Before word lists for German, Spanish, Italian, Portuguese, Dutch, Turkish and Arabic (connector 2.63.0), every unknown word in those languages was coded, so attributes (marital status, sexual orientation, offences, medical information, nationality, gender, job titles) were hidden by accident. They are now readable, as in French: Hidden Tokens codes identity and leaves attributes readable so the AI can work. REDACT counts them as personal data and some are special categories: that is a product choice, open to discussion, which we make visible here. See the tiers below.
- Connector 2.63.0 let through values that 2.57.0 coded. A German surname that is also a common noun, after a first name ("Lena Brunnen"); handles and usernames made of words; account and employee numbers after a label in a language without a word list. The table below counts them. Connector 2.64.0 fixes these cases with general rules: no value that 2.63.0 coded is left untouched. Against 2.57.0, 4 values are still left untouched that it coded (CS 3, HU 1): short numbers and a first name in languages without a word list, which it hid only by accident, because the word next to them was coded.
- Language detection in 2.64.0 is stricter (it needs a function word of the language) so that a Danish or Czech paragraph is no longer taken for German: a few short Spanish or Portuguese paragraphs are no longer recognised, and their over-coding goes back up by 1 to 5 points.
- Languages without a word list are still coded almost entirely: Nordic and Slavic languages, Hungarian, Hebrew, Hindi, Thai, Korean, Chinese and Japanese (which would first need word segmentation). Protected, but of little use to the AI.
- Decoding restores one spelling per code. A word coded once with a capital letter (at the start of a sentence, in a heading) comes back capitalised everywhere; an initial ("P. Haverkamp") disappears into the name. That is the "one person, one code" rule, and it is the first reason coded texts do not decode back exactly. It also explains the slight drop on the file route (765 → 751 of 1,000 from 2.57.0 to 2.64.0): a word now readable in lower case is recoded elsewhere once the same word has been coded with a capital letter, and comes back capitalised.
| Versions | Newly coded | Newly left untouched | Types newly left untouched |
|---|---|---|---|
| 2.57.0 → 2.63.0 | 41 | 15 | Employee_ID_Number (3), Social_Media_Identifiers (3), Building_Badge_Card_Number (3), National_Identification_Number (2), First_Given_Name (1), Customer_Reference_Number (1), Last_Family_Name (1), Account_Statements (1) |
| 2.63.0 → 2.64.0 | 35 | 0 | — |
| 2.64.0 → 2.77.0 | 0 | 0 | — |
| 2.77.0 → 2.77.0 + Double-check | 61 | 2 | First_Given_Name (1), Customer_Reference_Number (1) |
| 2.77.0 → 2.85.1 | 2 | 2 | Customer_Reference_Number (2) |
| 2.85.1 → 2.85.1 + Double-check | 56 | 0 | — |
| REDACT tier | touched / hidden, 2.57.0 | touched / hidden, 2.63.0 | touched / hidden, 2.64.0 | touched / hidden, 2.77.0 | touched / hidden, 2.77.0 + Double-check | touched / hidden, 2.85.1 | touched / hidden, 2.85.1 + Double-check |
|---|---|---|---|---|---|---|---|
| HIGH | 84.6% / 57.5% | 86.8% / 59.2% | 87.4% / 60.0% | 87.3% / 60.1% | 93.9% / 78.8% | 87.3% / 60.1% | 93.9% / 77.6% |
| MEDIUM | 98.0% / 89.1% | 97.7% / 89.7% | 97.9% / 90.2% | 97.9% / 90.2% | 98.2% / 95.8% | 97.9% / 90.2% | 98.2% / 95.1% |
| LOW | 79.8% / 59.1% | 77.5% / 58.4% | 77.8% / 58.9% | 77.9% / 59.1% | 80.4% / 62.8% | 77.8% / 59.1% | 80.8% / 62.9% |
Ordinary text coded, by language
The share of words that belong to no REDACT entity and that Hidden Tokens coded: over-coding, reversible, but it takes meaning away from the AI.
| Language | 2.57.0 | 2.63.0 | 2.64.0 | 2.77.0 | 2.77.0 + Double-check | 2.85.1 | 2.85.1 + Double-check |
|---|---|---|---|---|---|---|---|
| FR | 10.7% | 10.7% | 10.7% | 10.7% | 12.2% | 10.7% | 12.9% |
| EN | 24.6% | 23.8% | 24.1% | 23.8% | 25.4% | 23.6% | 27.2% |
| DE | 78.5% | 44.1% | 44.7% | 44.5% | 45.0% | 44.5% | 46.1% |
| ES | 65.6% | 39.2% | 44.5% | 44.5% | 43.2% | 44.5% | 54.2% |
| IT | 65.2% | 36.7% | 38.0% | 37.9% | 36.8% | 37.9% | 44.1% |
| PT_BR | 68.5% | 40.2% | 41.8% | 41.7% | 41.3% | 41.7% | 47.9% |
| PT_EU | 65.5% | 42.4% | 45.0% | 44.9% | 45.9% | 44.8% | 49.4% |
| NL | 61.9% | 43.2% | 43.7% | 43.5% | 44.5% | 43.5% | 46.2% |
| TR | 85.6% | 54.1% | 54.8% | 54.8% | 55.1% | 54.8% | 60.5% |
| AR | 76.9% | 64.6% | 64.6% | 64.4% | 65.2% | 64.4% | 69.9% |
Other languages (no word list yet)
| Language | 2.57.0 | 2.63.0 | 2.64.0 | 2.77.0 | 2.77.0 + Double-check | 2.85.1 | 2.85.1 + Double-check |
|---|---|---|---|---|---|---|---|
| CS | 73.7% | 71.9% | 72.0% | 71.8% | 72.0% | 71.8% | 72.4% |
| DA | 67.1% | 64.6% | 64.9% | 64.7% | 65.1% | 64.7% | 66.4% |
| FI | 78.1% | 76.0% | 76.0% | 76.4% | 76.7% | 76.4% | 78.2% |
| HE | 65.7% | 65.5% | 65.5% | 65.5% | 65.7% | 65.5% | 66.2% |
| HI | 51.1% | 48.5% | 48.7% | 48.4% | 49.6% | 48.4% | 52.1% |
| HU | 73.9% | 71.2% | 71.4% | 71.3% | 72.2% | 71.3% | 73.0% |
| JA | 74.3% | 68.8% | 68.8% | 68.7% | 68.8% | 68.7% | 69.4% |
| KO | 80.8% | 77.6% | 77.7% | 77.6% | 78.3% | 77.6% | 79.4% |
| NO | 72.6% | 70.1% | 70.2% | 70.2% | 71.4% | 70.2% | 71.8% |
| RU | 83.5% | 81.9% | 81.9% | 81.9% | 82.6% | 81.9% | 82.8% |
| SV | 73.0% | 71.6% | 71.6% | 71.5% | 72.3% | 71.5% | 72.9% |
| TH | 79.7% | 75.6% | 75.7% | 75.5% | 75.9% | 75.5% | 76.6% |
| ZH_CN | 77.4% | 76.0% | 76.0% | 75.9% | 78.9% | 75.9% | 79.3% |
| ZH_TW | 61.7% | 58.8% | 60.3% | 60.7% | 61.0% | 60.5% | 61.8% |
The official typed score
The authors' method: an entity is found if a prediction of the same type overlaps it (partial micro-F1), or has the same boundaries (exact). As the authors do for Presidio, each Hidden Tokens prefix maps to one type (TEL → Telephone_Numbers_Personal, a first name followed by a surname → Full_Name…); a code without a type (CODE-…, an unknown word) does not count. The five detectors' figures are those published by the authors.
| Detector | Partial micro-F1 | Macro F1 | Exact micro-F1 |
|---|---|---|---|
| Presidio | 0.195 | 0.063 | 0.145 |
| GLiNER-multi | 0.320 | 0.224 | 0.293 |
| OpenAI Privacy Filter | 0.512 | 0.254 | 0.171 |
| GPT-4.1 | 0.597 | 0.565 | 0.558 |
| Claude Sonnet 4.6 | 0.636 | 0.619 | 0.602 |
| Hidden Tokens 2.85.1 | 0.212 | 0.076 | 0.176 |
| Hidden Tokens 2.85.1 + Double-check | 0.223 | 0.102 | 0.189 |
Method and limits
- Data. REDACT's locked split,
pii_benchmark_sample1000.json(SHA-256754f562ce9c3c5b7b663f1ba6127c38a31d93f5631d38364341cb727936234c6), under the REDACT Dataset Terms. The dataset is not redistributed here. - Run. For each record, an empty key; the text goes through
anonymiser_fichier(a file) andcoder_texte(a message, with the output lock). The coded text is aligned on the original, then decoded the waydeanonymiserdoes. No settings, no forced language, no pack. The Double-check column: the same run withpasser.mjs --second-lecteurand the verified packagedouble-verification-1; each record keeps the number of passages added, read from the report, which shows the second reader fell back to the rules alone for none of them. - "Touched", "hidden". A value is touched if at least one of its letters or digits is under a code; hidden if all of them are. Only fully disclosed values count (
disclosed,complete). - Our measurement, not an independent audit. The failure classes fixed in connectors 2.62.0 and 2.64.0 were read on this very split: for them it is no longer blind. The fixes are general rules, checked on our own sentences. The next run will validate them on the 12,427 records outside the split.
- Synthetic. REDACT is made of generated documents. This is not a promise about your files; that is why Hidden Tokens shows a plan before coding and keeps a local log of everything the AI received.
Reproduce
Raw results: redact-2.57.0.json, redact-2.63.0.json, redact-2.64.0.json, redact-2.77.0.json, redact-2.77.0-dv.json, redact-2.85.1.json, redact-2.85.1-dv.json (every table, per type, per language, F5 and F6). Predictions (predictions-2.85.1.jsonl.gz, predictions-2.85.1-dv.jsonl.gz): positions and code prefixes for each record, with no dataset text: with REDACT and noter.mjs, every score on this page can be recomputed. Harness: README, passer.mjs, noter.mjs, comparer.mjs, types.mjs, alignement.mjs. Re-running the engine needs the connector source, which will be public with the open core.
Cite REDACT: Guneesh Vats, Anubha Agrawal, Shikha Singhal, Ajita Dash, Praison Selvaraj, Vidhan Jhawar, Ranga Prasad Chenna, Bharadwaj Y M G, "REDACT: A Systematically Controlled Multilingual Benchmark for Personal Information Detection", EMNLP 2026 Industry Track, arXiv 2606.19881.