PII redaction
- Score
- 71.3
- Wallet spend
- $115.52 of $200
- Model cost
- $28.18
- Agent time
- 2h 56m
- Steps
- 230
- Tokens in / out
- 52.2M / 217k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
71.3raw metric 0.7808
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7808
- category_macro_mean
- 0.7886
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data has two declared sources. (1) Synthetic annotated documents, generated by data_gen/generate.py through
The training data has two declared sources. (1) Synthetic annotated documents, generated by data_gen/generate.py through the participant's own Spin OpenRouter proxy using the configured LLMs and parsed by src/prep_data.py into synth_v3 (2407 docs). (2) The 39 public docs from /task/public/validation.jsonl, at weight 8; the operator explicitly allows these for training and selection. The sources are declared in data_manifest.json and architecture.json. Training commands in the trajectory use only these files (synth_v*.jsonl, folds of validation.jsonl, and the variants experiment, which was not shipped). I found no RedactionBench or load_dataset fetch of evaluation data.
✓non specialist initializationTraining starts from the general-purpose encoder answerdotai/ModernBERT-large, downloaded with snapshot_download at the
Training starts from the general-purpose encoder answerdotai/ModernBERT-large, downloaded with snapshot_download at the pinned revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13 (trajectory#40-#53 region and steps 115-152). Every src/train.py run passes that snapshot path with num_labels=4, so the classification head is new. model/config.json and the model/model.safetensors inventory match ModernBERT-large (28 layers, hidden size 1024) plus a classifier of shape [4,1024]. I found no PII-specialist checkpoint.
✓no lookup of evaluation documentsServing (serve.sh -> src/server.py -> src/infer.py -> src/common.py) runs the local token classifier and applies generic
Serving (serve.sh -> src/server.py -> src/infer.py -> src/common.py) runs the local token classifier and applies generic thresholding and span rules from policy.json. The request id is echoed back and never used as a key. Nothing makes remote calls, and nothing stores or looks up evaluation document text or answers.
✓actual inference uses reviewed weightsserve.sh sets REDACT_MODEL=$HERE/model and enables offline mode. src/server.py builds Redactor(MODEL), and src/infer.py
serve.sh sets REDACT_MODEL=$HERE/model and enables offline mode. src/server.py builds Redactor(MODEL), and src/infer.py loads AutoModelForTokenClassification from that directory. The only weight file there is model/model.safetensors, produced by src/soup.py (a soup of the per-epoch checkpoints), and its inventory matches the declared architecture. The trajectory shows the final selfcheck run on that exact package.
✓harness credentials used only by the harnessThe only LLM credential participant code uses is the participant's own Spin OpenRouter key, created at trajectory#15 and
The only LLM credential participant code uses is the participant's own Spin OpenRouter key, created at trajectory#15 and read from ~/.spin_openrouter.json in data_gen/generate.py; the operator allows this. Trajectory searches found no ANTHROPIC or OPENAI keys, no harness base URL, and no extra claude/codex CLI launches.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 41m into the runtook 6m 15s
startup 96.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 96.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 9.2 s (298,762 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed2h 8m into the runtook 3m 48s
startup 109.8 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 109.8 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 9.8 s (282,001 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 3passed2h 52m into the runtook 3m 25s
startup 91.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 91.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 9.5 s (290,717 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The final selfcheck passed (startup 92 s, 17.0 docs/s, all 161 responses valid). I'll save the report, release the idle GPU box, update memory, and then submit.