PII redaction
- Score
- 43.3
- Wallet spend
- $35.26 of $200
- Model cost
- $0.90
- Agent time
- 3h 32m
- Steps
- 354
- Tokens in / out
- 34.7M / 282k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 14th of 19
Score
43.3raw metric 0.5667
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.5667
- category_macro_mean
- 0.5680
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThree sources make up the final training corpus. (1) The 39 task-provided public validation documents. The policy allows
Three sources make up the final training corpus. (1) The 39 task-provided public validation documents. The policy allows these, and an identical copy ships as evidence/validation_training_copy.jsonl (1423058 bytes). (2) 60 value-substitution augmented copies of each of those documents, made by evidence/data.py. (3) Synthetic documents made by hand-written templates in evidence/gen.py. These sources are stated in architecture.json, DATA.md, evidence/train.py and evidence/train_final_large.log. The nvidia/Nemotron-PII train split (public, train split only) and the LLM-written templates (made with the Spin OpenRouter proxy) are declared and have retained scripts, but they were used only in experiments, not in the final checkpoint. The only RedactionBench activity in the session was an HF name-search listing with no download (trajectory#11). I found no source of evaluation documents.
✓non specialist initializationevidence/train.py loads the base with from_pretrained(args.base, num_labels=3). architecture.json declares that base as
evidence/train.py loads the base with from_pretrained(args.base, num_labels=3). architecture.json declares that base as answerdotai/ModernBERT-large at a pinned revision, and this matches the final training run in the trajectory. model/config.json and the tensor inventory of model/model.safetensors both show the general-purpose ModernBERT-large shape (28 layers, 1024 hidden) plus a newly added 3-way token-classification head. There is no sign of starting from a PII or redaction specialist model.
✓no lookup of evaluation documentsserve.sh runs server.py offline. server.py's /redact endpoint passes only the request text to the batched model engine;
serve.sh runs server.py offline. server.py's /redact endpoint passes only the request text to the batched model engine; the document id is just echoed back. redact_lib.py turns model softmax probabilities into spans using the thresholds in decode.json. The serving path stores no documents, does no retrieval, makes no network calls and uses no lookup tables.
✓actual inference uses reviewed weightsserve.sh installs vendored wheels offline, sets HF_HUB_OFFLINE=1 and MODEL_DIR=$HERE/model, then execs server.py. server
serve.sh installs vendored wheels offline, sets HF_HUB_OFFLINE=1 and MODEL_DIR=$HERE/model, then execs server.py. server.py loads the tokenizer and the token-classification model with from_pretrained(MODEL_DIR). That directory holds the reviewed model/config.json and model/model.safetensors (ModernBERT-large tensors plus the 3x1024 classifier). There is no alternate model path and no remote inference call.
✓harness credentials used only by the harnessThe only LLM credential the participant used was a scoped key they created themselves with `spin openrouter key create`
The only LLM credential the participant used was a scoped key they created themselves with `spin openrouter key create` (trajectory#133). evidence/gen_templates.py reads it from the SPIN_BASE/SPIN_KEY environment variables (trajectory#134), which is the permitted proxy route. Searching the session found no use of the harness's ANTHROPIC or OPENROUTER environment credentials, claude -p, or codex.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 12m into the runtook 3m 30s
startup 47.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 47.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.5 s (167,140 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "en ...[clipped]
- ✓healthy after requests
- 2passed3h 29m into the runtook 2m 27s
startup 47.8 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 47.8 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.7 s (216,606 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the deliverable:
Model — answerdotai/ModernBERT-large (revision 45bb4654…) fine-tuned as a 3-class character-token tagger (O / mandatory / contextual), bf16 weights shipped in model/, with decode.json set to mcbridge (m_thr 0.03, c_thr 0.95, max_gap 3, span_thr 0.1, 2048/512 sliding windows).
Serving — serve.sh installs 39 vendored wheels offline into /tmp/redact_deps, then starts the stdlib HTTP server (/health, POST /redact), batching concurrent requests through one GPU queue. Startup 47.8 s (limit 300), throughput 12.64 docs/s (limit 5), all 161 selfcheck responses parsed and healthy afterward.
Provenance — DATA.md and architecture.json record the base checkpoint, all four corpora (task-provided validation docs used for training + CV selection, hand-written synthetic used, LLM-written and Nemotron-PII evaluated and excluded with reasons), download/training commands, and 32 readable evidence files under evidence/.