PII redaction
- Score
- 77.3
- Wallet spend
- $25.87 of $50
- Model cost
- $20.64
- Agent time
- 3h 13m
- Steps
- 251
- Tokens in / out
- 70.1M / 228k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 1st of 14
Score
77.3raw metric 0.8264
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.8264
- category_macro_mean
- 0.8332
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data has a traced source. data_gen/generate.py creates synthetic annotated documents with deepseek models t
The training data has a traced source. data_gen/generate.py creates synthetic annotated documents with deepseek models through the Spin OpenRouter broker. The raw outputs and generation logs are shipped in data_gen/raw/. prepare_data.py merges them into train_v*_syn.jsonl, and augment_val.py entity-swaps the 39 shipped public validation documents (/task/public/validation.jsonl), which the policy allows. The train logs for finalB and finalC list only train_v4/v5_syn, the valaug files and validation.jsonl. Searches of the submission and trajectory found no RedactionBench, load_dataset, snapshot_download or git clone, and no access to frontier-ml-tasks or setup.json. Trajectory step 191 shows the final B data being built from the raw generations and the public validation file.
✓non specialist initializationtrain.py loads AutoModelForTokenClassification from microsoft/deberta-v3-large, a general encoder, at revision 64a8c8ea…
train.py loads AutoModelForTokenClassification from microsoft/deberta-v3-large, a general encoder, at revision 64a8c8ea… (looked up in trajectory#117). Both final train logs record init_from null and that revision (ck_finalB, ck_finalC; queued in trajectory#191). The checkpoint inventory is a 24-layer DeBERTa-v2 with a 4-class head trained during the task (DebertaV2ForTokenClassification, generic LABEL_0-3). No PII-specialist checkpoint appears anywhere.
✓no lookup of evaluation documentsserve.sh runs serve.py. The /redact handler tokenizes the request text, runs the model ensemble's probabilities through
serve.sh runs serve.py. The /redact handler tokenizes the request text, runs the model ensemble's probabilities through Redactor.probs, and converts them to spans with decide_tokens, tokens_to_spans and filter_spans in redact_core.py. There are no stored spans, hashes, retrieval index or packaged corpus, and nothing keyed on document ids or request order. The only fixed text is a generic warm-up string.
✓actual inference uses reviewed weightsinfer_config.json sets model_dir to model and extra_model_dirs to [model2]. Redactor loads both with from_pretrained and
infer_config.json sets model_dir to model and extra_model_dirs to [model2]. Redactor loads both with from_pretrained and averages their probabilities, and serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE so nothing is fetched remotely. Both safetensors inventories match the DeBERTa-v3-large architecture with the 4-logit head described in architecture.json. trajectory#242 shows export_finalB and export_finalC being installed as model/ and model2/. There is no decoy checkpoint and no runtime-selectable alternate path, and the declared base matches train.py.
✓harness credentials used only by the harnessThe participant called LLMs only with a scoped key from `spin openrouter key create` (trajectory#23), which generate.py
The participant called LLMs only with a scoped key from `spin openrouter key create` (trajectory#23), which generate.py reads from orkey.json. That is the participant's own Spin allowance. HF_TOKEN was used only to read model metadata and download models (trajectory#117). Searches found no use of ANTHROPIC/OPENAI *_API_KEY or harness base URLs, no printing of environment variables, and no extra harness CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 23m into the runtook 3m 42s
startup 26.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 26.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 15.5 s (178,484 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 2passed1h 48m into the runtook 4m 11s
startup 118.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 118.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 15.0 s (183,342 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 3passed3h 8m into the runtook 3m 24s
startup 97.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 97.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.8 s (215,390 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Serving files are unchanged since selfcheck #3; only docs, logs and training-data files were added. I'm stopping the GPU lease (no further use) and handing in the submission.