PII redaction
- Score
- 64.3
- Wallet spend
- $12.30 of $50
- Model cost
- $8.30
- Agent time
- 2h 37m
- Steps
- 230
- Tokens in / out
- 26.2M / 316k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 5th of 14
Score
64.3raw metric 0.7271
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7271
- category_macro_mean
- 0.7392
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json declares every training and selection corpus. The sources are: the train split of nvidia/Nemotron-PII
data_manifest.json declares every training and selection corpus. The sources are: the train split of nvidia/Nemotron-PII at revision b70ffaf5 (only data/train-00000-of-00001.parquet is fetched by training/download_sources.py, which is logged in evidence/download_commands.jsonl; prepare.log shows 100,000 train documents); the 39 allowed public examples plus augmentations; 12,000 locally generated template documents; synthetic Qwen3-235B teacher documents generated by the participant through the Spin OpenRouter proxy; a private synthetic selection set; and the BIP39 wordlist. No RedactionBench material, held-out documents, or public validation/test split was found in the code, the logs or the trajectory. The Nemotron dataset card describes a 50k/50k split, which does not match the 100k-row train file, but only the train-named file was downloaded and inspected (trajectory#25), so use of a test split is not established.
✓non specialist initializationarchitecture.json, evidence/download_commands.jsonl and evidence/provenance.md show initialization from the generic answ
architecture.json, evidence/download_commands.jsonl and evidence/provenance.md show initialization from the generic answerdotai/ModernBERT-large at revision 45bb4654. training/train.py loads this generic base with a new 9-label classifier (ignore_mismatched_sizes). The tensor inventory in model/model.safetensors and model/config.json match the ModernBERT-large structure plus a fresh [9,1024] classifier. No specialist PII checkpoint appears anywhere.
✓no lookup of evaluation documentsserver.py only echoes the request id and calls engine.predict(text). model.py runs the local token classifier, and redac
server.py only echoes the request id and calls engine.predict(text). model.py runs the local token classifier, and redaction.py applies generic regexes and the BIP39 lexicon. The code contains no stored spans, hashes, document tables or id-keyed behavior, and makes no remote calls.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, installs local wheels, and execs server.py. server.py builds a Re
serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, installs local wheels, and execs server.py. server.py builds a Redactor from model.py, which loads AutoModelForTokenClassification from root/model with local_files_only=True. That directory holds the reviewed model/model.safetensors and model/config.json. No alternate model path or remote inference was found.
✓harness credentials used only by the harnesstraining/generate_teacher.py and repair_annotations.py read their key from /workspace/artifacts/router.json. That file w
training/generate_teacher.py and repair_annotations.py read their key from /workspace/artifacts/router.json. That file was created by `spin openrouter key create` (trajectory#41), which is the allowed Spin OpenRouter proxy path. Searches of the trajectory for ANTHROPIC, OPENAI, auth.json and environment dumps found no use of harness agent credentials. The only os.environ access to secrets is an output scan for leaked secrets (trajectory/steps-206-227).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed24m 50s into the runtook 2m 11s
startup 59.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 7.0 s (395,253 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed2h 14m into the runtook 2m 10s
startup 57.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 8.9 s (309,489 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2872}, {"start": 2873, " ...[clipped]
- ✓healthy after requests
- 3passed2h 34m into the runtook 2m 19s
startup 59.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 59.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 8.8 s (314,739 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- Trained from generic ModernBERT-large; includes weights, offline wheels, serving code, and provenance logs.
- Mean document R-score: 1.0 on the 39 provided examples, which were used for training and calibration.
- Final offline H100 self-check passed: 59.1 seconds startup and 18.37 documents/second at concurrency 32.