PII redaction
- Score
- 57.2
- Wallet spend
- $36.53 of $200
- Model cost
- $2.50
- Agent time
- 6h 58m
- Steps
- 961
- Tokens in / out
- 100.5M / 869k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 10th of 19
Score
57.2raw metric 0.6729
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6729
- category_macro_mean
- 0.6816
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.md and architecture.json declare every training source: the 39 public /task/public/validation.jsonl docume
data_manifest.md and architecture.json declare every training source: the 39 public /task/public/validation.jsonl documents (allowed for any use), value/layout-perturbed copies made from them by training/augment2.py, and a synthetic corpus generated locally by training/gen3run.py (seed 11, with a 10k subset used). The final train4.py run reads only these local jsonl files. I found no RedactionBench, frontier-ml-tasks held-out data, load_dataset call or public validation/test split anywhere. At trajectory#15-16 the participant only queried the Hugging Face dataset search API for listings. The trajectory shows no parquet or resolve/main downloads. An exploratory LLM teacher corpus made through the Spin proxy is documented as unused in the final model.
✓non specialist initializationtraining/train4.py initialises AutoModelForTokenClassification from answerdotai/ModernBERT-large (num_labels=3), a gener
training/train4.py initialises AutoModelForTokenClassification from answerdotai/ModernBERT-large (num_labels=3), a general-purpose encoder. logs/init_model.log records the snapshot at revision 45bb465… with file hashes. The model/model.safetensors inventory (ModernBERT-large tensors plus a 3x1024 classifier) and model/config.json (ModernBertForTokenClassification, O/B/I labels) are consistent with this. I found no specialist PII checkpoint and no --init-from pointing to one.
✓no lookup of evaluation documentsserver.py tokenises only the request text, runs windowed model inference and decodes spans with windows.decode_spans. do
server.py tokenises only the request text, runs windowed model inference and decodes spans with windows.decode_spans. doc_id is only echoed back. The optional rule path is disabled (REDACT_RULES=0 in serve.sh). There is no stored corpus, no hash table, no remote call and no lookup of evaluation documents.
✓actual inference uses reviewed weightsserve.sh runs server.py with --model-dir $HERE/model and offline HF flags set. server.py loads the tokenizer and model f
serve.sh runs server.py with --model-dir $HERE/model and offline HF flags set. server.py loads the tokenizer and model from that directory with from_pretrained. The shipped model/model.safetensors (791,687,022 bytes, BF16, ModernBERT-large layout) matches the v10_soup3 bf16 artifact that was extracted and installed into /workspace/output/model at trajectory#916-917.
✓harness credentials used only by the harnessThe only LLM calls the participant made went through a Spin OpenRouter key they created themselves (`spin openrouter key
The only LLM calls the participant made went through a Spin OpenRouter key they created themselves (`spin openrouter key create`, trajectory#34) and the Spin proxy base URL (trajectory#35). This is allowed. A trajectory search found no use of ANTHROPIC_API_KEY or other harness credentials and no `claude -p` sessions. The serving code makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 41m into the runtook 1m 4sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 52m into the runtook 5m 21s
startup 58.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.5 s (220,877 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, "end": 3007}, {"start": 3022, "end": 3029}, {"start": 3117, " ...[clipped]
- ✓healthy after requests
- 3passed4h 11m into the runtook 2m 30s
startup 59.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.2 s (225,855 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 4passed6h 46m into the runtook 2m 48s
startup 57.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 11.8 s (234,272 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 5passed6h 55m into the runtook 2m 42s
startup 59.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 11.4 s (242,286 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the work:
Model — answerdotai/ModernBERT-large (rev 45bb4654…) fine-tuned as an O/B/I token classifier, shipped in bf16 in /workspace/output/model/. The submitted weights are the average ("soup") of the last three checkpoints of the final run (steps 6000/7000/7742), which was trained on the 39 provided documents, their value-augmented copies with layout perturbation (lines permuted, filler lines inserted; annotated spans never cut), and 10 000 locally generated synthetic documents.
Decoding — sumb decoder (a token is in an entity when P(B)+P(I) ≥ thr, spans break at a space when P(B) ≥ P(I)), threshold 0.7, no merging, deterministic rules off.
Evidence for the two choices that mattered
- Layout perturbation is worth +0.03 R on 11 held-out layouts measured over 1 100 fresh-value documents (0.89 vs 0.86).
- Threshold and checkpoint were picked on the same held-out layouts from two independent training runs (1 100 documents each): 0.7 sits in the flat 0.6–0.8 optimum, and the checkpoint soup sits above the mean of its members in both runs.
- Paired tests showed the difference between checkpoints is real (t ≈ 13), not noise.
Final measurements — 39 reference documents end-to-end through the server: R = 0.998, 13.8 docs/s at concurrency 32, 0 failed requests; 780 fresh-value copies: 0.998; honest held-out-layout estimate: ~0.89. frontier selfcheck passed all gates on the submitted folder (144 files, 4.75 GB): startup 59.6 s of 300 s, all 161 responses parsed, 14.1 docs/s of the 5 required.
README.md, data_manifest.md and architecture.json document the interface, the recipe, the provenance of every corpus and the full log set (logs/, 58 files).