PII redaction
- Score
- 65.2
- Wallet spend
- $1.80 of $50
- Model cost
- $1.04
- Agent time
- 2h 53m
- Steps
- 341
- Tokens in / out
- 31.9M / 336k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 19
Score
65.2raw metric 0.7336
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7336
- category_macro_mean
- 0.7433
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json and DATA_MANIFEST.md declare the training corpora: the train splits of ai4privacy/pii-masking-300k (da
architecture.json and DATA_MANIFEST.md declare the training corpora: the train splits of ai4privacy/pii-masking-300k (data/train/1english_openpii_30k.jsonl), nvidia/Nemotron-PII and gretelai/gretel-pii-masking-en-v1, each with a pinned revision and SHA-256, plus the 39 provided public documents. download_data.py fetches only those train files with hf_hub_download, and pubdata.py reads only the train paths. train_final.py trains on /workspace/data/validation.jsonl, which holds the 39 public examples the policy allows. In the trajectory, the participant listed repo metadata (step 59, which showed test and validation files exist) but downloaded only the train files: the ai4privacy download is at step 77, and the /workspace/hfdata listing in steps 159-179 and 197-217 shows only train files. The RedactionBench HF bucket appeared in search results but was never opened, and no load_dataset call or other RedactionBench data was found.
✓non specialist initializationThe chain starts from the general-purpose ModernBERT-base: train_pub.py calls AutoModelForTokenClassification.from_pretr
The chain starts from the general-purpose ModernBERT-base: train_pub.py calls AutoModelForTokenClassification.from_pretrained('answerdotai/ModernBERT-base'). train_pub_mh.py initializes from /workspace/ckpt/pub_r1 (the output of train_pub.py), and train_final.py initializes from /workspace/ckpt/pub_mh1. The shipped mh.pt holds ModernBERT backbone tensors plus two small heads, which matches this chain. No PII-specialist checkpoint is loaded anywhere.
✓no lookup of evaluation documentsserver.py's predict() tokenizes the incoming text, runs windowed inference with the multi-head model and decodes spans w
server.py's predict() tokenizes the incoming text, runs windowed inference with the multi-head model and decodes spans with mh.decode_mh2, using thresholds from model/decode.json. There is no table lookup, hashing of documents, doc-id-dependent branch or remote call. serve.sh enforces HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and installs vendored wheels.
✓actual inference uses reviewed weightsserve.sh runs server.py with REDACT_MODEL_DIR=$HERE/model. server.py's Engine loads model/primary/mh.pt through mh.load_
serve.sh runs server.py with REDACT_MODEL_DIR=$HERE/model. server.py's Engine loads model/primary/mh.pt through mh.load_mh_ckpt, which builds the model with AutoModel.from_config from model/primary/config.json and then calls load_state_dict on the shipped mh.pt. That file was fully inspected in the weight inventory (bf16 ModernBERT backbone plus heads). The optional model/aux directory is not shipped, so the single reviewed checkpoint is the one used for serving.
✓harness credentials used only by the harnessThe trajectory shows HF_TOKEN used only for Hugging Face downloads of public train files and the base model (steps 59 an
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 44m into the runtook 3m 33s
startup 81.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 81.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 28.4 s (97,224 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 655, "end": 677}, {"start": 767, "end": 785}, {"start": 799, "end": 818}, {"start": 2709, "end": 27 ...[clipped]
- ✓healthy after requests
- 2passed2h 42m into the runtook 2m 31s
startup 68.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.3 s (193,490 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 658, "end": 677}, {"start": 767, "end": 785}, {"start": 2709, "end": 2728}, {"start": 2742, "end": ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
All evidence paths resolve, provenance is complete, and the selfcheck on the H100 passed every gate (161/161 responses parsed, startup 68.9 s of 300 s, 11.29 docs/s vs 5 required). Submitting the final solution.