Hidden Tokens

Hidden Tokens benchmark: detection per industry pack, and leak test

We don't ask you to trust a percentage. Here is what Hidden Tokens detects in each industry, measured on documents it was never tuned on, how it compares with Presidio, what changed in the latest version, what it still misses, and how to check every number yourself.

Updated 25 October 2026 · HT-Bench 0.2.0, test split · connector 2.28.0

HT-Bench: what is detected, pack by pack

HT-Bench is our detection benchmark. It feeds synthetic documents of each industry (call-centre transcripts, court submissions, payslips, medical reports, bank statements, invoices, KYC files) through the Hidden Tokens engine with that industry's pack switched on, and checks every labelled value: coded, partly coded, or left readable. The numbers below come from one run on 25 October 2026, connector 2.28.0, on the test split: documents generated with their own seed, which no engine rule has been tuned on.

Direct identifiers are names, ID card, passport and residence-permit numbers, NIR, CNSS, ICE, IF, RC, SIREN and SIRET, phone numbers, emails, IBAN and RIB, IMEI and IMSI, card numbers and security codes. "Recall, all types" adds the other labelled values (addresses, company names, dates of birth, BIC, case, invoice, cheque, policy and claim numbers…). Precision is the share of coded passages that cover a labelled value; it drops when the engine codes ordinary words. The "no pack" row is general documents processed without an industry pack.

What changed from 2.27.0 to 2.28.0

In 2.27.0, 3 packs were below 99% because of a handful of formats. Connector 2.28.0 fixes each of them:

• French social security numbers written with a dash before the key ("… 407 - 86"): the key was left readable. The engine now reads the key after a dash, and only if it is the right key for the number.
• Seven-digit CNSS employer affiliation numbers on Moroccan payslips, when a word such as "employeur" sits between the acronym and the number.
• Moroccan RIBs and French IBANs written in groups of four: their first 16 digits sometimes pass the card-number check, so the end stayed readable. Bank account numbers are now recognised before card numbers.
• Names with a particle ("de La …", "Le …", "Ould …") and names with an apostrophe: the last word, or the part after the apostrophe, stayed readable.
• Damaged output in healthcare documents: some pack rules matched inside longer words (a medicine name, "Lotissement"), which produced nested codes such as NOM-001-002. Pack rules now match whole words only, and every one of the 366 coded documents decodes back to the exact original.

These failures were read and fixed on the development split. Nothing was tuned on the test split, but you should know two things: its 2.27.0 gaps were published on this page before the fix, and they were the same formats, because both splits are built from the same document templates. So the test split shows that the fixes hold on values and phrasings the engine was not tuned on; it does not show how Hidden Tokens handles formats nobody has written a template for yet.

What is still missed

All labelled values together: 98.7% detected, 44 values left readable, all of them outside the direct-identifier target: cheque or bill (LCN) number 75.0% (12 of 48 missed), date of birth 79.1% (18 of 86 missed), postal address 92.4% (14 of 184 missed). The output lock, which re-checks what the AI receives through coder_texte, still masks some codes already placed by a pack (24 in HR and payroll), shown as "•••": the value is protected, but the AI cannot quote the code.

Per entity type: direct identifiers

* Precision per type is counted by code prefix, and some types share one: the CIN prefix is also used for NIR and passports, the RIB prefix for IBAN and RIB, the SIREN prefix for SIRET and generic references. For those types a per-type precision would be meaningless, so it is not shown (it is in the raw JSON); the pack precision above is the measure of over-coding. For names, the 80.1% precision reflects the engine coding unknown capitalised words as names: protective, but it gives the AI less to read.

Compared with Presidio

We ran Microsoft Presidio on the same 366 documents and scored it with the same code (lib/score.mjs): any span Presidio returns counts as a coded passage. Configuration: presidio-analyzer 2.2.358 as shipped, AnalyzerEngine with its default recognisers, spaCy 3.8.7 with en_core_web_lg 3.8.0 and fr_core_news_lg 3.8.0, each document analysed in English and in French, union of the spans, no score threshold: the most favourable setting for its recall. It ran on the same computer with sockets and DNS disabled before Presidio was imported.

• Direct identifiers: Hidden Tokens 100.0%, Presidio 77.5%. All labelled values: 98.7% against 68.7%.
• Presidio's default recognisers include none for Moroccan formats (CIN, ICE, RIB), and it does not tag company names (1.8% recall). What it catches on these documents comes largely from recognisers built for other countries' formats (US driver's licence, Indian PAN, US bank number…), which flag almost any code or long number. That is how it reaches 100% on CNSS, IF and IMSI, and why it coded 37.6% of the decoys.
• Presidio is a toolkit meant to be extended with your own recognisers. This compares its defaults with a finished product, not with what an engineering team could build on Presidio.

Method

• Synthetic data only. Names are common Moroccan, Maghrebi and French first names and surnames recombined at random, in Latin script. Numbers are drawn at random and completed with their check digits (Luhn, mod 97…): they have the shape of a real number and designate no one. No real file, not even an "anonymised" one, and no network call.
• Held-out test split. HT-Bench 0.2.0 has two splits from the same templates. We read gaps and tune the engine on the development split (seed ht-bench-v0). The test split (seed ht-bench-v0-test, SHA-256 4881c8d2bc9febe584e4dab351739c8bd00858a20eab834c15993fcace79e10b) has other values and other phrasing; fewer than 1% of its identifiers also appear in the development split. Every published number comes from the test split. If a rule were ever written after looking at a test miss, we would change the test seed and say so.
• Size. 366 documents, 3,420 labelled values (2,523 direct identifiers) and 3,029 decoys that must stay readable.
• What is measured. Each document goes through the real connector over the Model Context Protocol (anonymiser_fichier on a text file, the same coding as coder_texte and PDF), with the document's pack enabled as in the app and a fresh key for each document. We read the coded file written on the computer, realign it with the original, and check that decoding it with its key gives back the original exactly.
• What counts as a hit. A value is detected only if every letter and digit of it is under a code. Name particles (Ben, El, Aït, Ould, de, Le…) and, in addresses, short numbers and structure words (n°, rue, Douar…) may stay readable. A partly coded value is a miss. Recall counts a value as protected whatever code prefix it gets. A decoy counts as wrongly coded as soon as one of its characters is coded. F1 = 2PR / (P + R).

Limits

• Our own benchmark, on synthetic documents. It is not an independent audit, and 100.0% here is not a promise about your files. Synthetic text is cleaner than real files: no OCR errors, no broken layouts, no handwriting, no formats outside our templates. That is why Hidden Tokens shows a plan before coding and keeps a local log of everything the AI received.
• Same templates in both splits. The test split checks unseen values and phrasings of the formats we model, not unseen formats.
• Text only. Excel table mode and Word documents are not scored by HT-Bench (the leak test below covers them from another angle). One pack per document, no firm-wide policy.
• Arabic script is not covered, only Latin transliterations of Arabic names.
• Notary pack not measured. The notariat-ma pack ships with the connector, but HT-Bench 0.2.0 has no notary documents yet.

Reproduce it

Everything is in the public ai4x-apps repository, commit 205cbba (connector 2.28.0). The run takes about half a minute; the Presidio comparison about one minute.

git clone https://github.com/ai4xedu/ai4x-apps && cd ai4x-apps
git checkout 205cbba42de7feaa01d20430ddd6523d4f970163
cd apps/mcp-anonymiseur && npm ci
node bench/ht-bench/generer.mjs --verifier   # the documents match their seeds
npm run bench:ht:test                        # → bench/ht-bench/resultats/test/RAPPORT.md

# Optional, Presidio: a separate Python environment, installed once (the only step that uses the network)
cd bench/ht-bench/comparaison
uv venv -p 3.12 .venv && VIRTUAL_ENV=.venv uv pip install -r requirements.txt "preshed==3.0.9"
cd ../../.. && npm run bench:ht:comparer     # → bench/ht-bench/resultats/test/comparaison.md

We pin preshed to 3.0.9 because version 3.0.14, the one installed by default in October 2026, stops spaCy 3.8.7 from loading its models. It does not change any result.

Raw results: ht-bench-2.28.0.json (JSON, about 190 KB): versions, seed, fingerprint, every table on this page and the full output of both runs. Cite this file for these numbers. Each version that changes detection gets its own file, and earlier ones stay online: ht-bench-2.27.0.json.

Leak test: what the AI can still read

HT-Bench measures detection value by value. The leak test asks a different question: after Hidden Tokens has coded a folder, can the AI still read any sensitive value? We don't look at what Hidden Tokens thinks it detected. We collect everything Claude could read and search it for every real value. Run on connector 2.22.0, 24 October 2026.

• The corpus: 15 fictional client documents written to look like real case files: employment-tribunal submissions, a commercial-lease summons, client meeting notes, a formal notice, a witness statement, a list of exhibits, an English client email, an invoice, a payroll sheet, an unpaid-invoices list, a loan agreement, an audit note mixing French and Moroccan Arabic, a card-payment journal, a chargeback dispute and a vehicle-fleet list. Formats: Markdown, Excel and PDF; the same run is repeated with the text documents as Word files.
• The ground truth: 201 sensitive values listed by hand, in 11 categories.
• What the AI can see: the harness drives the real Hidden Tokens connector through the Model Context Protocol, exactly as Claude Desktop does: list the folder, show the plan, code file by file, then code the whole folder. It gathers every tool response, every coded file (content, file name, sheet names) and the batch report: 49 outputs in total.
• What counts as a leak: a value is leaked if it appears once, anywhere. Numbers are compared digits only, so "06 48 27 19 53" and "0648271953" are the same value.

Results

The two columns differ only on amounts. In "auto" mode, the 11 amounts left readable were invoice lines, receivables and card-payment totals; the payroll figures were coded. That is the behaviour we chose after testing with clients: a bare invoice amount does not identify anyone, and the AI needs it to build totals and charts. If your figures are confidential, set the folder to "protect" once in the app and every number becomes a consistent placeholder, restored on your computer afterwards.

Keeping the document usable

Coding everything would be easy and useless: the AI would receive a wall of codes. So we also measure what should stay readable.

• Domain terms that must stay readable (courts, legal sources, products, statuses): 52 of 56 kept (92.9%). The four lost were "Conseil de prud'hommes", "Tribunal judiciaire", a van model and the phrase "non attribuée".
• Ordinary words across all documents: 548 of 634 kept readable (86.4%).

Known limits of the leak test

• English text is over-coded. In the English client email, only 11 of 30 ordinary words stayed readable: words such as "Dear", "please" or "Friday" were coded. Nothing leaked, but the AI has less to work with. The engine was built first for French and Moroccan documents; English is our priority fix.
• Lists of places and brands in spreadsheets (shops, car models) are often coded when they could stay readable: 57.7% and 50% of ordinary words kept in the two worst files.
• This is our own benchmark on fictional data, not an independent audit. It shows the method and the trend between versions; it does not prove that every real file will be handled perfectly.

The harness (bench/run.mjs) and the fictional corpus (bench/corpus.mjs) are in the ai4x-apps repository. Run node bench/run.mjs for the default setting, with ANX_CHIFFRES=gabarit to protect all amounts, or with --word to feed the text documents as Word files.

On your own files, you don't need our word either: the local "What Claude saw" page re-checks every response the AI received against your key. See proof, not promises.