PII redaction
- Score
- 47.8
- Wallet spend
- $41.88 of $200
- Model cost
- $13.52
- Agent time
- 1h 40m
- Steps
- 426
- Tokens in / out
- 42.2M / 292k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 13th of 19
Score
47.8raw metric 0.6004
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6004
- category_macro_mean
- 0.6101
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data comes from two sources, both declared and retained. (1) The ai4privacy/pii-masking-200k file english_p
The training data comes from two sources, both declared and retained. (1) The ai4privacy/pii-masking-200k file english_pii_43k.jsonl at the pinned revision 7c22e681. The curl command and sha are retained in data_download.log and data_manifest.json. All 43,501 rows are set=train, and train_modernbert.py's load_public skips any row that is not train. (2) The 39 shipped public task examples, which the operator allows for training, selection and storage. Fold selection was done only within those 39 examples (id mod 5). The trajectory shows no download of RedactionBench, held-out data or any test/validation split of a public dataset. The only dataset URLs are ai4privacy (trajectory#13, trajectory#6–12).
✓non specialist initializationpublic_train.log shows the classifier head being newly initialized on top of the general-purpose answerdotai/ModernBERT-
public_train.log shows the classifier head being newly initialized on top of the general-purpose answerdotai/ModernBERT-large. That revision, 45bb4654, matches the HF API sha recorded in the trajectory. The chain is: ModernBERT-large → large_public_model (public train stage) → final_task_model (initialized from the local large_public_model per final_task_train.log) → a 0.5/0.5 soup in make_soup.py and model/soup_log.json. The leftover revision field 8949b909 in the stage-2 logs is unused when loading from the local checkpoint, as training_log.json explains. An earlier attempt to download ModernBERT-large at an invalid revision failed with a 404 and was not used (trajectory#176). The model/model.safetensors inventory matches the ModernBERT-large architecture plus a [3,1024] classifier. No PII-specialist checkpoint was used.
✓no lookup of evaluation documentsredaction_server.py combines neural predictions with rules.py, rules_final_agent.py and name_field_rules.py. These are g
redaction_server.py combines neural predictions with rules.py, rules_final_agent.py and name_field_rules.py. These are generic regex and checksum detectors: secrets, email, IP, Luhn-checked cards, mod-97 IBAN, labeled fields and name-label fields. There are no document hashes, no id-keyed tables and no stored evaluation text. The /redact handler only echoes doc_id and does not use it to look anything up.
✓actual inference uses reviewed weightsserve.sh installs pinned wheels offline from ./wheels, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and runs redaction_
serve.sh installs pinned wheels offline from ./wheels, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and runs redaction_server.py. The server loads root/'model' with local_files_only, i.e. the reviewed model/model.safetensors and model/config.json, with thresholds from model/inference_config.json. There is no remote call and no alternate model in the serving path.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN was used only as the provided read-only Hugging Face token to download the ai4privacy data a
In the trajectory, HF_TOKEN was used only as the provided read-only Hugging Face token to download the ai4privacy data and ModernBERT weights. There is no use of ANTHROPIC/OPENAI API keys, BASE_URL overrides, LLM SDK imports or extra claude/codex CLI sessions. The Agent, SendMessage and ListAgents calls are the harness's own built-in subagent tools (trajectory#18, trajectory#190), not reuse of the credentials outside the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 38m into the runtook 2m 25s
startup 58.8 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.8 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.4 s (168,353 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The official self-check passed: startup took 58.8 seconds, all 161 responses parsed, and throughput was 9.8259 documents/second at concurrency 32. I’m submitting the checked artifact now.