PII redaction
- Score
- 73.1
- Wallet spend
- $8.22 of $10
- Model cost
- $19.24
- Agent time
- 3h 53m
- Steps
- 225
- Tokens in / out
- 60.4M / 195k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
73.1raw metric 0.7941
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7941
- category_macro_mean
- 0.8122
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, calibration and selection corpus is declared in a shipped file. architecture.json, training/DATA_MANIFES
Every training, calibration and selection corpus is declared in a shipped file. architecture.json, training/DATA_MANIFEST.md and training/COMMANDS.md list three sources: the 39 shipped public samples (data/val_all.jsonl, weighted x3, also used for 3-fold out-of-fold threshold calibration), about 2123 synthetic documents in training/data/syn_v4.jsonl, and the base weights. training/gen.py made the synthetic documents through the allowed Spin OpenRouter proxy, using faker seeds and few-shot examples taken from the provided samples. The trajectory shows the participant fetched RedactionBench's HF API metadata and downloaded only its README card (trajectory#18). No parquet, loader or mirror was downloaded or used: searches for load_dataset, datasets-server and parquet downloads found nothing. The card is not evaluation data. I found no held-out data and no validation or test split of any public dataset. The generation runs are backed by retained logs and commands.
✓non specialist initializationtraining/train.py builds RedactTagger(AutoModel.from_pretrained(a.model)), with the default set to answerdotai/ModernBER
training/train.py builds RedactTagger(AutoModel.from_pretrained(a.model)), with the default set to answerdotai/ModernBERT-large. COMMANDS.md runs snapshot_download of ModernBERT-large at the pinned revision recorded in architecture.json. ModernBERT-large is a general-purpose encoder, not a PII model. The tensor inventory of model/encoder/model.safetensors matches ModernBERT-large: 50368x1024 embeddings, Wqkv 3072x1024 and Wi 5248x1024. model/heads.pt contains only the new head_c (3x1024) and head_w (2x1024) heads. The trajectory's model searches (trajectory#21) covered general encoders only, and I found no PII-specialist checkpoint (such as GLiNER-PII) anywhere.
✓no lookup of evaluation documentsserve.py tokenizes the request text, runs the encoder and heads over windows, decodes spans with decode_doc3 from redact
serve.py tokenizes the request text, runs the encoder and heads over windows, decodes spans with decode_doc3 from redact_model.py, and applies only threshold settings from model/serve_config.json. It contains no regex or answer tables, no hashing of documents, no stored evaluation documents and no remote calls. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE.
✓actual inference uses reviewed weightsserve.sh runs `exec python serve.py`. serve.py's load_model() loads model/encoder with AutoModel and model/heads.pt into
serve.sh runs `exec python serve.py`. serve.py's load_model() loads model/encoder with AutoModel and model/heads.pt into RedactTagger with a strict key check, and the /redact endpoint runs those weights. The shipped inventories match the architecture: a ModernBERT-large encoder plus head_c and head_w tensors that match RedactTagger. training/export.py produced these artifacts from the training checkpoint.
✓harness credentials used only by the harnessThe only credential the participant created and used was the Spin OpenRouter key from `spin openrouter key create` (traj
The only credential the participant created and used was the Spin OpenRouter key from `spin openrouter key create` (trajectory#28), which training/gen.py reads from .orkey.json to call the Spin proxy; this is allowed. I found no use of harness credentials in the code or the session. The ANTHROPIC_API_KEY-like strings in the synthetic data are fake values the generator produced, and no CLI worker launched with harness keys (such as claude -p) appears.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 17m into the runtook 2m 3s
startup 29.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 29.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 10.5 s (262,883 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 2passed3h 50m into the runtook 2m 19s
startup 36.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 36.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.9 s (214,114 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything's documented. Releasing the GPU (it expires shortly anyway) and running a second self-check on the final folder to confirm its digest: