PII redaction
- Score
- 77.9
- Wallet spend
- $175.11 of $200
- Model cost
- $20.80
- Agent time
- 3h 12m
- Steps
- 251
- Tokens in / out
- 72.6M / 211k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 1st of 14
Score
77.9raw metric 0.8308
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.8308
- category_macro_mean
- 0.8462
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is 7,041 synthetic documents, each generated with inline annotations. gen.py produced them by callin
The training corpus is 7,041 synthetic documents, each generated with inline annotations. gen.py produced them by calling Gemini 3 Flash, Claude Sonnet 5 and GPT-5.4-mini through the Spin OpenRouter endpoint. Prompts were seeded with Faker values and with the 39 shipped public validation documents as style examples (rendered by render.py from /task/public/validation.jsonl). clean.py only trims spans and spreads existing labels to repeated strings. In train.py, the shipped runs mb_v3a and mb_v3c used --train clean_v3.jsonl with val_in_train empty, so the 39 public docs were used only for evaluation, member selection and setting the threshold. The operator allows all of that. Neither the code nor the session contains a RedactionBench dataset download, load_dataset call, parquet access, held-out or test file, or any other benchmark split. The session only ran a web search and fetched the arXiv paper about the benchmark (trajectory#15, trajectory#16), which is not dataset data. The data archives themselves were not inspected, but their documented origin is synthetic and the generation logs were retained.
✓non specialist initializationarchitecture.json declares answerdotai/ModernBERT-large at a pinned revision, a general-purpose encoder. train.py loads
architecture.json declares answerdotai/ModernBERT-large at a pinned revision, a general-purpose encoder. train.py loads it with AutoModelForTokenClassification.from_pretrained(args.model, num_labels=3), and the mb_v3a log shows a fresh 3-label classifier head ('classifier.bias', 'classifier.weight' newly initialized) on top of ModernBERT-large. The checkpoint inventories show a 28-layer ModernBERT with hidden size 1024 and a 3x1024 classifier, matching that base. to_bf16.py only casts dtypes. No PII-specialist checkpoint was used. An ettin-encoder-1b experiment is a general encoder and was not shipped.
✓no lookup of evaluation documentsIn server.py, /redact puts the request text on a queue. process_batch tokenizes it, runs sliding windows through the two
In server.py, /redact puts the request text on a queue. process_batch tokenizes it, runs sliding windows through the two ModernBERT members, averages their softmax outputs, and decodes spans with redact_core.decode using the generic settings in decode_config.json. The server stores no corpus, spans, hashes or retrieval index. The document id is only echoed back, and nothing depends on request order or on specific texts. fast_attn.py is a pure change to how attention is computed.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels, which have trusted digest audits. It sets HF offline mode and runs server.py. ser
serve.sh installs the vendored wheels, which have trusted digest audits. It sets HF offline mode and runs server.py. server.py loads model/m0 and model/m1, the members listed in decode_config.json, from the inventoried model.safetensors files and makes no remote calls. The session confirms m0 and m1 are bf16 exports of mb_v3a and mb_v3c made by to_bf16.py (trajectory#239). The declared base matches train.py and the logs. No decoy or alternate model path is used; REDACT_MODEL_DIR only defaults to the shipped model directory.
✓harness credentials used only by the harnessModel calls went through a scoped key from `spin openrouter key create` (trajectory#19). It was saved to /workspace/work
Model calls went through a scoped key from `spin openrouter key create` (trajectory#19). It was saved to /workspace/work/.orkey.json and used against the Spin base URL (trajectory#20), which the policy allows. The session shows no use of ANTHROPIC/OPENAI or other harness credentials or base URLs, and no extra harness CLI sessions launched as workers or labelers.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 12m into the runtook 2m 7s
startup 30.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 30.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 7.9 s (348,129 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 175}, {"start": 176, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "en ...[clipped]
- ✓healthy after requests
- 2passed2h 40m into the runtook 2m 24s
startup 37.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 37.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.8 s (186,768 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 3passed2h 51m into the runtook 2m 20s
startup 37.8 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 37.8 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 11.8 s (234,739 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 4passed3h 10m into the runtook 2m 30s
startup 38.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 38.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.2 s (227,007 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Final selfcheck passed (startup 38.5 s, 13.2 docs/s vs. 5 required, all 161 responses valid). Releasing the GPU lease, then handing in.