PII redaction
- Score
- 69.7
- Wallet spend
- $3.79 of $10
- Model cost
- $13.48
- Agent time
- 1h 50m
- Steps
- 298
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 4th of 19
Score
69.7raw metric 0.7684
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7684
- category_macro_mean
- 0.7757
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, calibration and selection corpus is declared in architecture.json and data_manifest.json. They are: the
Every training, calibration and selection corpus is declared in architecture.json and data_manifest.json. They are: the nvidia/Nemotron-PII train parquet, the Podric/prowl-secrets-corpus corpus.parquet (train), the Hodfa71/pstu-synthetic-secrets secrets_train.jsonl (all at pinned revisions), synthetic docs built in code from the public BIP-39 wordlist, and the 39 public /task/public/validation.jsonl docs plus entity-swap augmentations, which operator policy allows. src/download_data.sh and src/download_data.log keep the pinned download commands and their results. src/prepare_data.py reads only these sources. In the trajectory, the Nemotron test parquet, the prowlbench eval files and the ai4privacy/gretel repos were only listed during exploration (trajectory#38). None was downloaded or used. Only the declared train files were fetched (trajectory#40, trajectory#119). There is no reference to RedactionBench, held-out task data or any validation/test split of a public dataset.
✓non specialist initializationsrc/train.py starts Stage 1 from the general-purpose AnswerDotAI/ModernBERT-large at the pinned revision 45bb4654… with
src/train.py starts Stage 1 from the general-purpose AnswerDotAI/ModernBERT-large at the pinned revision 45bb4654… with a fresh 5-label token-classification head. Stage 2 and Stage 3 continue from that lineage. src/build_merged_soup.py averages the full, soup and stage1 checkpoints from that same lineage. The weight inventory of models/merged_soup_model/model.safetensors shows a standard ModernBERT-large structure (50368x1024 embeddings, Wqkv/Wi layers) with a 5x1024 classifier. The trajectory shows the stage1, fold and full models being produced by the participant's own training runs (trajectory#119, trajectory#196) and merged at trajectory#270. No GLiNER, Piiranha, ai4privacy or other PII-NER checkpoint is loaded.
✓no lookup of evaluation documentsserver.py and decoder.py run model inference followed by generic regex and threshold post-processing. They contain no st
server.py and decoder.py run model inference followed by generic regex and threshold post-processing. They contain no stored document corpus, no id or text hash lookup and no network calls (serve.sh sets HF_HUB_OFFLINE=1). The only labeled documents used are the 39 public validation docs, which operator policy allows. No RedactionBench, frontier-ml-tasks held-out data or mirrors appear anywhere in the code or the trajectory.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py reads models/ensemble.json, which lists only merged_soup_model with weight 1.0, and c
serve.sh runs server.py. server.py reads models/ensemble.json, which lists only merged_soup_model with weight 1.0, and calls AutoModelForTokenClassification.from_pretrained on models/merged_soup_model. That is the reviewed safetensors file in the weight inventory. /redact calls engine.infer_document: sliding-window softmax averaging from this model, then decode_document_spans using models/decoder_config.json. The fallback model directories do not ship. The trajectory confirms the merge wrote merged_soup_model and ensemble.json into the output (trajectory#270) and that the final model was loaded from the output directory for testing.
✓harness credentials used only by the harnessSearches of the trajectory for API_KEY, BASE_URL, claude, codex and other agent CLIs found no use of harness credentials
Searches of the trajectory for API_KEY, BASE_URL, claude, codex and other agent CLIs found no use of harness credentials by participant code and no launch of other agent CLIs. The only 'gemini' matches are the harness's own antigravity-cli log paths. Remote work used the permitted spin compute and spin S3 transfer tooling (trajectory#247, trajectory#251). Hugging Face access was download-only, with no uploads to the Hub.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 46m into the runtook 2m 21s
startup 28.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 28.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 9.7 s (283,748 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests