PII redaction
- Score
- 58.9
- Wallet spend
- $12.40 of $50
- Model cost
- $6.78
- Agent time
- 2h 56m
- Steps
- 305
- Tokens in / out
- 110.0M / 445k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
58.9raw metric 0.6857
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 157
- document_mean
- 0.7031
- category_macro_mean
- 0.7141
- metric
- mean_document_r_score
- scored_items
- 157
- failed_items
- 4
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training data is the 39 shipped public examples (/task/public/validation.jsonl, which policy allows) plus 40,000 doc
The training data is the 39 shipped public examples (/task/public/validation.jsonl, which policy allows) plus 40,000 documents converted from the train splits of three public datasets: Nemotron-PII, Gretel and ai4privacy. code/make_public_pii.py reads only train-00000 parquet/jsonl train files. The provenance scripts call hf_hub_download for train files only, and their logs, manifests, pinned revisions and checksums match. In the trajectory, step 74 only lists repo filenames (including validation/test names) without downloading them, and step 76 downloads only the train files. I found no use of RedactionBench, validation/test splits or the held-out set.
✓non specialist initializationBoth served models were fine-tuned from the generic MLM checkpoint answerdotai/ModernBERT-large at a pinned revision (pr
Both served models were fine-tuned from the generic MLM checkpoint answerdotai/ModernBERT-large at a pinned revision (provenance/base_model_download.md, code/run_final_all.sh, code/train_tagger.py). The F1 training log shows the base loaded with classifier.weight/bias MISSING and decoder.bias UNEXPECTED, which fits a generic MLM base rather than a PII-specialist checkpoint. The weight inventories hold standard ModernBERT tensors with a new 3-class head. A DeBERTa-v3-large base was downloaded only for an experiment that is not served.
✓no lookup of evaluation documentscode/server.py reads only req['text'] and echoes the id. It tokenizes the text, runs the model ensemble in windows, aver
code/server.py reads only req['text'] and echoes the id. It tokenizes the text, runs the model ensemble in windows, averages P(mandatory), and decode_spans (code/common.py) turns that into spans over the request text. There is no stored corpus, hash table, remote call or retrieval of evaluation documents. serve.sh sets the HF and Transformers libraries to offline mode.
✓actual inference uses reviewed weightsserve.sh runs code/server.py with code/serving_config.json, which lists models/F1 and models/F2. The Engine loads each o
serve.sh runs code/server.py with code/serving_config.json, which lists models/F1 and models/F2. The Engine loads each one with AutoModelForTokenClassification.from_pretrained and uses its softmax outputs for every prediction. The shipped models/F1 and models/F2 model.safetensors inventories match the export manifest (size and dtype), and the trajectory shows the exported F1/F2 being copied into the output tree. The wheels are approved upstream digests.
✓harness credentials used only by the harnessThe trajectory shows the participant using the provided spin compute and s5cmd tooling for GPU training, uploads and ass
The trajectory shows the participant using the provided spin compute and s5cmd tooling for GPU training, uploads and assembling the output tree. Searches found no reading, printing, copying or use of harness API credentials (ANTHROPIC/API_KEY/BASE_URL hits were only in documentation or argparse help text) and no extra harness agent sessions launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 46m into the runtook 3m 19s
startup 63.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 63.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 13.8 s (200,367 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed2h 52m into the runtook 3m 18s
startup 68.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.1 s (195,341 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and graded as it stands in /workspace/output. It's a two-model ensemble of ModernBERT-large token classifiers that tags mandatory spans, served by serve.sh.
Results
- Cross-validation (leave-fold-out over the 39 documents, scored with the task's
spans.py): a single model with public data reaches about 0.67–0.70 document mean at threshold 0.3. Without public data it was about 0.56. A two-model ensemble is about 0.69–0.70, and three members about 0.71. The final models were trained on all 39 documents, so they have no held-out estimate of their own. - Official selfcheck, run twice on the final folder, both passed:
- Startup: 63–68 s against a 300 s limit.
- Throughput: 11.4–11.7 docs/s against a required 5 (161 documents in about 14 s, with 32 s allowed). No failed or malformed responses.
What I chose and why
- Two members instead of three. Three members measured 6.9 docs/s on an H100 SXM, which is too close to the gate. The third member was worth about 0.01 in CV, within noise. I dropped it.
- Stride 1024 instead of overlapping windows. Single-model CV was the same as stride 512 (0.698 vs 0.697), and it halves the compute.
- Grouped batching in the server. Queued documents share GPU batches. Identical spans to the one-at-a-time path on the tiny model, and in-process on the real models, except one document that differs by bf16 rounding.
- Threshold 0.3, the best value on both single-model and ensemble CV.
Training data and provenance
- Your 39 validation documents, plus mandatory-only documents from the training splits of Nemotron-PII, Gretel's English PII set, and the English ai4privacy file. No validation or test split of any public corpus was used.
- Download and conversion commands, logs, manifests, dataset revisions, and wheel checksums are in
provenance/. The base checkpoint is pinned to45bb4654…inarchitecture.json.
Things to know
- The selfcheck report says the architecture review did not run and can still score the submission zero. The evidence lists in
architecture.jsonall resolve to files in the folder. - The mandatory/contextual distinction is inconsistent across documents (for example, names on public records are contextual). That is the main limit on the score, and more data would be the lever.
- The final models have no held-out score. The 0.99 in-sample figure only shows that the pipeline runs.
- About $12.40 of the $50 wallet was spent. The GPU leases are terminated.