PII redaction
- Score
- 72.2
- Wallet spend
- $36.43 of $50
- Model cost
- $32.40
- Agent time
- 7h 13m
- Steps
- 432
- Tokens in / out
- 42.8M / 383k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 2nd of 14
Score
72.2raw metric 0.7871
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7871
- category_macro_mean
- 0.7991
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe shipped code and DATA_MANIFEST.md declare every training, calibration and selection corpus: (1) the 39 public valida
The shipped code and DATA_MANIFEST.md declare every training, calibration and selection corpus: (1) the 39 public validation documents (/task/public/validation.jsonl), which the policy allows for any use, and (2) LLM-synthetic documents the participant generated with training/gen.py through the Spin OpenRouter proxy (deepseek-v3.2 / gemini-2.5-flash). The synthetic files were kept in data/synthetic as jsonl files with generation logs. A Faker augmentation was used only in an ablation. The final training logs (logs/final/fx1.log and the others) show final=True, real_rep=4, syn_final=synA.pkl,synB.pkl, which matches the manifest. Searches found no use of a public benchmark validation or test split (no load_dataset, no RedactionBench, no hf datasets download) and no access to held-out task data. The only /task reads in the trajectory are the public validation file, the contract and the scoring helper.
✓non specialist initializationarchitecture.json declares answerdotai/ModernBERT-large at the pinned revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13,
architecture.json declares answerdotai/ModernBERT-large at the pinned revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13, a general-purpose encoder. The trajectory confirms snapshot_download of that revision. training/train_cv.py calls AutoModel.from_pretrained on that base and adds a freshly initialized nn.Linear head (normal init, std 0.02). The weight inventories for soupX and soupY contain only backbone.* ModernBERT tensors and head.weight [4,1024] / head.bias [4]. No PII/NER-specialist checkpoint was found anywhere.
✓no lookup of evaluation documentsIn serve/server.py, the /redact handler calls ENGINE.redact(text), and the request id is only echoed back. Spans come on
In serve/server.py, the /redact handler calls ENGINE.redact(text), and the request id is only echoed back. Spans come only from model softmax/sigmoid outputs followed by select_tokens and decode_spans. The server has no regex lookup tables, no stored document texts or answers, and no remote calls. serve.sh sets HF_HUB_OFFLINE=1.
✓actual inference uses reviewed weightsserve.sh execs serve/server.py. The server loads the files listed in model/serve_config.json (soupX.safetensors and soup
serve.sh execs serve/server.py. The server loads the files listed in model/serve_config.json (soupX.safetensors and soupY.safetensors) from MODEL_DIR with safetensors load_file. MODEL_DIR defaults to the shipped model/ directory, and serve.sh does not set the REDACT_MODEL_DIR override. Both files are in the weight inventory with the expected ModernBERT plus 4-way head structure. The trajectory shows they were built by training/make_soup.py from the final fx/fy runs and moved into the output model directory. No decoy or alternate inference path was found.
✓harness credentials used only by the harnessThe participant's LLM calls (training/llm.py) used a key created with `spin openrouter key create`, which points to the
The participant's LLM calls (training/llm.py) used a key created with `spin openrouter key create`, which points to the allowed Spin proxy base_url (api.spin.inductionlabs.com). Searches of the trajectory found no use of harness API keys (ANTHROPIC, OPENROUTER_API_KEY environment variable) and no harness CLI calls such as `claude -p`.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 19m into the runtook 3m 4s
startup 73.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 10.4 s (265,317 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 91, "end": 94}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 517, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 662, "end": 677}, {"start": 678, "end": 681}, {"start": 780, "end": 781}, ...[clipped]
- ✓healthy after requests
- 2passed5h 39m into the runtook 3m 15s
startup 78.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 78.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 18.8 s (146,349 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 2709, "end": 2728}, {"start": 2742, "end": 2757}, {"start": 2772, "end" ...[clipped]
- ✓healthy after requests
- 3passed7h 9m into the runtook 3m 38s
startup 78.8 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 78.8 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 18.9 s (146,119 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 2709, "end": 2728}, {"start": 2742, "end": 2757}, {"start": 2772, "end" ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is over. The last self-check passed on an H100 with 8.5 docs/s against the required 5, and startup in 79 s against the 300 s limit.
The server runs two weight-averaged groups of ModernBERT-large models and averages their predictions. One group averages 5 fine-tuned runs and the other 6. A governor drops to a single group if the backlog drains slower than 6.5 docs/s. On the one-GPU test (self-check) the governor never needed it, and the grading budget is 32.2 s for 161 held-out documents.
My expected score is roughly 0.82–0.84 mean R. That comes from cross-validation on the 39 validation documents, with run-to-run noise of about ±0.015. A model trained on real documents only scored 0.66, so most of the gain comes from the synthetic training data.
Per-document scores are uneven. The long AWS-log and file documents (validation-0024, 0025) score only around 0.4–0.5 in cross-validation.
Provenance files are in /workspace/output (architecture.json, DATA_MANIFEST.md, training/RUN_COMMANDS.md, logs/selection_results.txt). The architecture review did not run in any self-check, so I have not seen how it judges them.