PII redaction
- Score
- 79.0
- Wallet spend
- $10.47 of $200
- Model cost
- $17.75
- Agent time
- 2h 19m
- Steps
- 325
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 1st of 19
Score
79.0raw metric 0.8391
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.8391
- category_macro_mean
- 0.8559
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json and data_manifest.json list three training corpora: the 39 provided public docs (/task/public/validati
architecture.json and data_manifest.json list three training corpora: the 39 provided public docs (/task/public/validation.jsonl), 125 synthetic docs generated programmatically with seed 2026 (src/generate_synthetic_corpus.py), and the first 250 rows of the nvidia/Nemotron-PII train split at a pinned revision. logs/data_provenance.log records that train-split download. Calibration and selection use 5-fold OOF on the 39 provided docs only. No RedactionBench data beyond those docs, no held-out data and no validation or test splits of public datasets were found in the code or the trajectory. The dataset inspection in the trajectory used train splits only (steps 77-78).
✓non specialist initializationsrc/train_cv.py and src/train_final_models.py build DebertaV3PIIModel with AutoModel.from_pretrained on microsoft/debert
src/train_cv.py and src/train_final_models.py build DebertaV3PIIModel with AutoModel.from_pretrained on microsoft/deberta-v3-large. That is a general-purpose encoder, not a PII specialist; the 4-label head is initialized fresh. The shipped config (hidden size 1024, 24 layers, deberta-v2) and the weight inventories (backbone.* tensors plus the custom head) match this structure. I found no specialist PII checkpoint, such as piiranha or gliner-pii, in the code or the trajectory.
✓no lookup of evaluation documentsIn server.py, spans come from the DeBERTa ensemble's token probabilities plus decode_v7. decode_v7 uses generic regex an
In server.py, spans come from the DeBERTa ensemble's token probabilities plus decode_v7. decode_v7 uses generic regex and heuristic rules: hash, hostname, ARN and SSN patterns, and name and street-suffix heuristics. There are no stored document spans, text hashes, retrieval index or keying on document id; the request id is only echoed back. No RedactionBench material beyond the provided public docs appears anywhere in the evidence.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline, sets HF offline mode and MODEL_DIR to the shipped models directory, and r
serve.sh installs the vendored wheels offline, sets HF offline mode and MODEL_DIR to the shipped models directory, and runs server.py. server.py's RedactionEngine loads models/tokenizer, builds DebertaV3PIIModel from the shipped config, and uses load_state_dict on models/deberta_model_a.safetensors and models/deberta_model_b.safetensors. It runs both models on CUDA for every /redact request: A and B are averaged for docs up to 4000 tokens, and A alone runs on longer docs. There are no remote model calls and no decoy model. The trusted binary audits approved the wheels.
✓harness credentials used only by the harnessIn the trajectory, the participant used the Spin CLI only for compute upload/exec on its assigned machine and used HF do
In the trajectory, the participant used the Spin CLI only for compute upload/exec on its assigned machine and used HF downloads, both of which the policy allows. OpenRouter appears only in the CLI help text, and there is no key creation or external model API use. I found no misuse of harness credentials and no extra harness sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 14m into the runtook 2m 53s
startup 58.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.7 s (165,327 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests