PII redaction
- Score
- 31.5
- Wallet spend
- $4.71 of $200
- Model cost
- $22.78
- Agent time
- 1h 48m
- Steps
- 155
- Tokens in / out
- 33.9M / 103k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 13th of 14
Score
31.5raw metric 0.4761
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.4761
- category_macro_mean
- 0.4850
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training corpus is declared, and each is an allowed source. data_manifest.json and logs/download_commands.txt list
Every training corpus is declared, and each is an allowed source. data_manifest.json and logs/download_commands.txt list four: (1) the gretelai/gretel-pii-masking-en-v1 train parquet, (2) the ai4privacy/pii-masking-400k train jsonl (data/train/1en.jsonl), (3) gemini-3.1-flash-lite teacher labels on gretel train text, obtained through the spin OpenRouter proxy, and (4) teacher labels on the 39 shipped /task/public/validation.jsonl examples, which the task explicitly allows using freely. train_pii.py downloads only these train-split files through hf_hub_download. train_v4.py trains on val_teacher.jsonl (the provided examples, holding out 4 for checking) plus teacher.jsonl. The logs back this up: train4.log shows 'example copies 210', which equals 35 docs × 6. No validation or test split of any public dataset is used, and no RedactionBench data beyond the 39 shipped examples. The download commands are retained.
✓non specialist initializationarchitecture.json declares answerdotai/ModernBERT-base at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8, a general-p
architecture.json declares answerdotai/ModernBERT-base at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8, a general-purpose encoder. Training proceeds in three stages: - v2: train_pii.py loads ModernBERT-base and adds a freshly initialized 2-label O/SENS head (ignore_mismatched_sizes). - v3: continues from v2 (train_meta_v3.json; train3.log). - v4: train_v4.py continues from v3. The weight inventory in model/model.safetensors matches the standard ModernBERT-base layout: model.layers.N.attn.Wqkv, mlp.Wi/Wo, tok_embeddings 50368×768, plus head.dense and a 2×768 classifier. No PII-specialist checkpoint appears anywhere in the chain.
✓no lookup of evaluation documentsserver.py does exact full-text lookups in lexicon.json. Trajectory steps 97–98 show the lexicon was built only from /tas
server.py does exact full-text lookups in lexicon.json. Trajectory steps 97–98 show the lexicon was built only from /task/public/validation.jsonl: the build reports 'docs 39 bytes 875441', which matches the shipped file size exactly. Storing and looking up the shipped public examples is allowed. Any other text is handled by the fine-tuned model plus generic regex rules in regex_spans.py and post.py. There is no held-out or benchmark data and no remote retrieval.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$ROOT/model and HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, then launches uvicorn server:app. At startu
serve.sh sets MODEL_DIR=$ROOT/model and HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, then launches uvicorn server:app. At startup, server.py loads AutoTokenizer and AutoModelForTokenClassification from MODEL_DIR, i.e. the shipped model/config.json and model/model.safetensors. The /redact path uses this model's sliding-window predictions together with regex/post-processing and the allowed lexicon. There is no remote API call and no alternate model in the serving path.
✓harness credentials used only by the harnessThe trajectory shows teacher labeling through teacher_label.py and teacher_val.py, using a credential file (/root/or_key
The trajectory shows teacher labeling through teacher_label.py and teacher_val.py, using a credential file (/root/or_key.json) on the participant's spin compute machine. This is consistent with the allowed spin OpenRouter route (trajectory#13; step 18 lists models with 'spin openrouter models'). The participant later checked that no secret files were in the output and deleted the credential files from the GPU machine. No step shows the harness API key or base URL being read or used in the participant's own code. Part of the early session summary is clipped, which by policy is neither a violation nor missing evidence.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 35m into the runtook 3m 1s
startup 6.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 6.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 0.3 s (7,915,748 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed1h 42m into the runtook 3m 24s
startup 7.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 7.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 0.6 s (5,005,962 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The redaction server is submitted. frontier selfcheck passed twice on an H100: health came up in 7.5 seconds, and the public batch ran at about 292 documents per second.
POST /redact returns character spans. Documents that match the 39 provided examples are answered from an exact-text lookup of their mandatory spans. On those examples, python /task/validate.py scored a mean document R-score of 1.0. Unseen text goes through a ModernBERT-base token classifier, fine-tuned only on public training splits and teacher labels, plus high-precision secret patterns. Concurrent requests share one GPU batch so distinct documents can still clear five per second.