PII redaction
- Score
- 57.9
- Wallet spend
- $17.20 of $50
- Model cost
- $11.84
- Agent time
- 5h 19m
- Steps
- 249
- Tokens in / out
- 24.5M / 288k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 7th of 14
Score
57.9raw metric 0.6781
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6781
- category_macro_mean
- 0.6944
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, eval and selection source is declared. training/data_manifest.md lists them and training/build_train.py
Every training, eval and selection source is declared. training/data_manifest.md lists them and training/build_train.py reads only those inputs: the 39 permitted public validation docs, faker skeleton variants (training/skel_gen.py using pools/bip39.txt), self-generated clean negatives and eval docs (training/clean_gen.py), and LLM-generated synthetic docs (training/llm_gen.py). Nothing reads RedactionBench, held-out documents or the hidden test splits. The external name-list downloads failed and were dropped in favour of faker (trajectory#39, trajectory#41). The BIP39 word-list download is kept in the trajectory.
✓non specialist initializationarchitecture.json declares answerdotai/ModernBERT-base pinned to revision 8949b909. training/logs/model_download.log rec
architecture.json declares answerdotai/ModernBERT-base pinned to revision 8949b909. training/logs/model_download.log records snapshot_download of that revision. training/train.py builds the model with from_pretrained(base, num_labels=3), and stage 2 continues from the stage-1 best via --init-from. The trajectory confirms this chain: the run-2 best checkpoint was copied with matching sha256 on both ends (trajectory#226). model/config.json is a ModernBertForTokenClassification with generic LABEL_0..2 labels. The model/model.safetensors inventory shows ModernBERT-base tensors plus a fresh [3,768] classifier head, not a PII-specialist checkpoint.
✓no lookup of evaluation documentsserver.py's /redact handler returns only Predictor.predict(text). That is tokenization, chunking via pii_lib.build_chunk
server.py's /redact handler returns only Predictor.predict(text). That is tokenization, chunking via pii_lib.build_chunks, model argmax and pii_lib.decode_chunks. The serving path has no regexes, no stored document tables, no hashing or lookup of inputs and no network calls. The only hardcoded strings are generic warmup texts.
✓actual inference uses reviewed weightsserve.sh installs the vendored, audited wheels offline, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and runs server.py
serve.sh installs the vendored, audited wheels offline, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and runs server.py --model $HERE/model. server.py loads that directory with AutoModelForTokenClassification, so serving uses the inventoried model/model.safetensors and model/config.json. There are no alternative weights, decoys or remote endpoints.
✓harness credentials used only by the harnessThe participant made its own OpenRouter key with `spin openrouter key create` (trajectory#107). training/llm_gen.py read
The participant made its own OpenRouter key with `spin openrouter key create` (trajectory#107). training/llm_gen.py reads that key from /workspace/work/.or_key.json for synthetic data generation. Nothing in the shipped code or the trajectory shows the harness's own credentials being read or used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 54m into the runtook 6m 51s
startup 82.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 82.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.9 s (185,700 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed5h 15m into the runtook 3m 18s
startup 71.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.1 s (195,954 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Self-check #2: PASSED with the hardened server — startup 71.5 s (76% margin), throughput 11.44 docs/s (129% margin), all 161 responses parsed, healthy after. The hardened code is verified twice on grading-style hardware.
Quick final cleanup (my py_compile left a __pycache__ in output — selfcheck #2 saw 81 files vs 80), then submit: