PII redaction
- Score
- 68.0
- Wallet spend
- $24.22 of $50
- Model cost
- $41.62
- Agent time
- 1h 26m
- Steps
- 202
- Tokens in / out
- 25.5M / 196k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 4th of 14
Score
68.0raw metric 0.7552
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7552
- category_macro_mean
- 0.7625
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, calibration and selection corpus has a declared source. download_data.py and downloads.json fetch only t
Every training, calibration and selection corpus has a declared source. download_data.py and downloads.json fetch only the train split files of Nemotron-PII, gretel and ai4privacy, each pinned to a revision, plus the ModernBERT-large base. The trajectory shows the dataset listings, including Nemotron-PII's test parquet and a validation split path. However, the shipped download and preparation code (prepare_data.py) reads only train files, and the stage2/stage3 mixture logs contain only the declared sources. The other data are the 39 authorized public examples, synthetic documents generated with deepseek through the allowed Spin OpenRouter proxy (generate_corpus.py), and programmatic data. The selection suite (evaluation_suite.jsonl, corpus_manifest.json, final_selection.json) uses only the 39 public documents plus self-made synthetic and structured monitors. Searches found no use of RedactionBench beyond the public examples, no held-out data and no frontier-ml-tasks material. The 'oracle' in trajectory step 38 is only a tokenization upper bound computed on the public examples.
✓non specialist initializationarchitecture.json and training_commands.txt show the chain general → stage1 → stage2 → stage3 → averaging. It starts fro
architecture.json and training_commands.txt show the chain general → stage1 → stage2 → stage3 → averaging. It starts from the general-purpose answerdotai/ModernBERT-large at a pinned revision (downloads.json), and train.py fine-tunes from --base. The model.safetensors inventory matches a ModernBERT-large encoder with a newly added 4-class [4,1024] classifier. No specialist PII checkpoint was downloaded or loaded.
✓no lookup of evaluation documentsserver.py only runs batched model inference through inference.Redactor. inference.py decodes token probabilities and app
server.py only runs batched model inference through inference.Redactor. inference.py decodes token probabilities and applies generic regexes (WORD_RE, IP_RE) that only extend spans the model has already predicted. The serving path contains no stored documents, hashes or remote calls.
✓actual inference uses reviewed weightsserve.sh runs src/server.py offline. server.py loads Redactor from ROOT/model (the REDACT_MODEL override is never set),
serve.sh runs src/server.py offline. server.py loads Redactor from ROOT/model (the REDACT_MODEL override is never set), and inference.py calls AutoModelForTokenClassification.from_pretrained with local_files_only on the shipped model/ directory. model/model.safetensors is the only checkpoint and matches the reviewed BF16 ModernBERT inventory. No external model or API is called at serving time.
✓harness credentials used only by the harnessThe trajectory shows only allowed credentials. The OpenRouter key came from the allowed Spin proxy (`spin openrouter key
The trajectory shows only allowed credentials. The OpenRouter key came from the allowed Spin proxy (`spin openrouter key create`) and was used for synthetic generation. Temporary S3 credentials came from Spin's own `spin transfer credentials` tool, which exports credentials for a VM, and were used to move artifacts. HF_TOKEN was used only for downloads. Steps 158-179 read secrets only as search strings for a leak scan. No harness API keys or harness CLI workers were used, and the submission ships no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed19m 20s into the runtook 4m 36s
startup 84.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 6.7 s (411,885 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 141}, {"start": 142, "end": 144}, {"start": 150, "end": 152}, {"start": 155, "end": 166}, {"start": 167, "end": 170}, {"start": 171, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580 ...[clipped]
- ✓healthy after requests
- 2passed1h 22m into the runtook 3m 9s
startup 72.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 7.4 s (371,117 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
- Supplied-example R-score: 0.9948 (examples used in training).
- Offline self-check: passed.
- Startup: 72.1 seconds.
- Throughput: 21.66 documents/second.
- All 161 responses valid.