PII redaction
- Score
- 41.1
- Wallet spend
- $11.51 of $200
- Model cost
- $6.94
- Agent time
- 1h 14m
- Steps
- 171
- Tokens in / out
- 35.6M / 161k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 11th of 14
Score
41.1raw metric 0.5491
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.5491
- category_macro_mean
- 0.5533
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training corpus has a declared source. Stage 1 streams only the 'train' splits of ai4privacy/pii-masking-300k and
Every training corpus has a declared source. Stage 1 streams only the 'train' splits of ai4privacy/pii-masking-300k and ai4privacy/pii-masking-200k at pinned revisions (training/data_common.py). The download logs fetch only data/train files. A tree listing of data/validation appears, but nothing from it was downloaded. Stages 2 and 3 use the 39 permitted docs in /task/public/validation.jsonl plus self-generated synthetic secrets and negatives (training/train_stage2.py, training/train_stage3.py, training/synth_neg.py). DATA_MANIFEST.json declares all of these with revisions and commands. No document matched 'RedactionBench', and no trajectory step matched 'frontier-ml-tasks'. The one web search (step 30) was a generic query that found no dataset, and nothing from it was used. Several other candidate datasets were probed in step 29 but did not exist and were not used.
✓non specialist initializationStage 1 loads the general-purpose base model answerdotai/ModernBERT-base at pinned revision 8949b909ec900327062f0ebf497f
Stage 1 loads the general-purpose base model answerdotai/ModernBERT-base at pinned revision 8949b909ec900327062f0ebf497f51aef5e6f0c8 with a fresh 3-label head (training/train_stage1.py). logs/stage1-train.log confirms this. Stage 2 starts from stage1/final and stage 3 from stage2/final, as the logs show. The model/config.json and model/model.safetensors inventory show a standard ModernBertForTokenClassification (22 layers, hidden size 768, 3x768 classifier), consistent with that chain. The trajectory (steps 31-32) shows only ModernBERT-base and deberta-v3-base being considered; no specialist PII checkpoint (e.g. piiranha, gliner) was loaded.
✓no lookup of evaluation documentsapp.py's /redact endpoint calls redactor.predict_spans(req.text), which runs model inference and a single JWT regex (red
app.py's /redact endpoint calls redactor.predict_spans(req.text), which runs model inference and a single JWT regex (redact_lib.py). There are no stored document spans, hashes, ids, retrieval indexes or remote calls. serve.sh runs offline. The extra regexes and the seed finder are not used on the serving path.
✓actual inference uses reviewed weightsserve.sh runs `uvicorn app:app` with MODEL_DIR defaulting to ./model. app.py builds redact_lib.Redactor(MODEL_DIR), whic
serve.sh runs `uvicorn app:app` with MODEL_DIR defaulting to ./model. app.py builds redact_lib.Redactor(MODEL_DIR), which calls AutoModelForTokenClassification.from_pretrained(model_dir) and applies threshold 0.3 to the PII probabilities. The weights loaded are therefore the shipped model/model.safetensors with model/config.json, whose inventory matches the declared architecture.
✓harness credentials used only by the harnessThe trajectory shows no use, echo or copy of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL variables. Searches f
The trajectory shows no use, echo or copy of ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or *_BASE_URL variables. Searches for those names found nothing; 'API_KEY' matches only synthetic template strings. No extra harness CLIs were launched. Compute ran through the spin CLI (permitted). HF_TOKEN was only prefix-checked and used for downloads; 'upload' commands were spin compute uploads of the participant's own scripts, not Hugging Face uploads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 1m into the runtook 2m 46s
startup 69.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 19.6 s (140,636 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed1h 10m into the runtook 2m 24s
startup 69.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 21.6 s (127,957 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Model: ModernBERT-base fine-tuned in 3 stages (ai4privacy train splits → provided validation mandatory-only → validation + synthetic negatives), token classifier with threshold 0.3, whitespace-safe decoding, single-space merging for multi-word entities, JWT-only regex backup.
Validation R-score 0.991 on provided 39 docs (GPU eval). Selfcheck PASSED twice: startup ~69s/300s, throughput 7.5-8.2 docs/s (req 5), all responses valid.
Files in /workspace/output/: serve.sh, app.py, redact_lib.py, model/, wheels/, architecture.json, DATA_MANIFEST.json, logs/, training/.