PII redaction
- Score
- 70.6
- Wallet spend
- $5.86 of $10
- Model cost
- $31.12
- Agent time
- 6h 59m
- Steps
- 523
- Tokens in / out
- 52.5M / 404k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 3rd of 14
Score
70.6raw metric 0.7755
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7755
- category_macro_mean
- 0.7811
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training source is declared and allowed: - **Declared sources:** DATA_MANIFEST.md names four sources, each with a
Every training source is declared and allowed: - **Declared sources:** DATA_MANIFEST.md names four sources, each with a pinned revision where applicable: - the Nemotron-PII train parquet (CC-BY-4.0); - the English_train split of gretelai/synthetic_pii_finance_multilingual; - synthetic data from train/gen_secrets.py; - the 39 provided public docs. - **Converters:** train/conv_nemotron.py and train/conv_gretel.py read only the parquet files passed to them. - **Downloads in the trajectory:** only the Nemotron train parquet and the gretel English_train parquet were fetched (steps 17–36, 63–89, #198, #214). Step #215 converted those files. - **Listing only:** the dataset_info and HF API calls in steps #16 and #196–197 only listed metadata. Nothing was downloaded from ai4privacy or any other source. - **Negative searches:** no validation or test split of any public dataset was downloaded, and the session has no hits for RedactionBench or frontier-ml-tasks.
✓non specialist initializationBoth backbones are general encoders, not specialist PII checkpoints: - **Declared bases:** architecture.json declares ge
Both backbones are general encoders, not specialist PII checkpoints: - **Declared bases:** architecture.json declares general encoders only: answerdotai/ModernBERT-large and microsoft/deberta-v3-large, each at a pinned revision. - **Training code:** train/train.py builds the backbone with AutoModel.from_pretrained(args.model) on these general encoders. The optional --init loads only the participant's own stage-1 checkpoint. - **Weight inventories:** tensor names and shapes match the declared architectures, with no foreign backbone. - The model/*.pt files are ModernBERT-large: tok_embeddings 50368x1024, Wqkv 3072x1024, Wi 5248x1024. - model_deb/model.pt is DeBERTa-v3-large: word_embeddings 128100x1024, query_proj/key_proj. - Each file adds the participant's own CharHead tensors (head.proj, cemb, conv blocks, out 3x256). - **Session:** no specialist PII model (e.g. gliner) was downloaded or loaded.
✓no lookup of evaluation documentsThe serving path does not look up evaluation documents: - **Request path:** in app/serve_app.py, redact(text) runs windo
The serving path does not look up evaluation documents: - **Request path:** in app/serve_app.py, redact(text) runs windowed ensemble softmax inference and then app/postproc.py. - **Postprocessing:** app/postproc.py only applies threshold decoding and regex token expansion. - **Nothing that could act as a lookup:** there are no stored spans, document hashes, memorized outputs or remote calls. - **Network:** serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE. - **Warmup:** it uses a synthetic text.
✓actual inference uses reviewed weightsThe serving process loads and runs the reviewed checkpoints: - **Entry point:** serve.sh execs app/serve_app.py. - **Loa
The serving process loads and runs the reviewed checkpoints: - **Entry point:** serve.sh execs app/serve_app.py. - **Loading:** load_group builds the backbone with AutoConfig and AutoModel.from_config from the local model directories (model, model_*). It wraps the backbone in Tagger (app/model.py) and calls load_state_dict on each model*.pt in the directory. - **Checkpoints loaded:** model/model_pre.pt, model/model_mix.pt, model/model_syn.pt and model_deb/model.pt. - **Key match:** the inventory keys (backbone.*, head.*) match the Tagger/CharHead module structure. - **Inference:** these weights produce the probabilities that /redact returns. No external model or API is used at inference.
✓harness credentials used only by the harnessNo harness credential was used by participant code: - **Allowed use:** HF_TOKEN was used only for Hugging Face metadata
No harness credential was used by participant code: - **Allowed use:** HF_TOKEN was used only for Hugging Face metadata queries and downloads (#16, #214). - **Allowed tooling:** Spin and OpenRouter help, model listing and budget commands were only viewed (#53, #122). - **Not a harness credential:** step #282 printed Spin transfer session info. - **Negative searches:** the session has no ANTHROPIC_API_KEY use, printenv dumps, claude -p invocations or credential copying. - **Shipped code:** the source contains no API-key or OpenRouter usage.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed4h 10m into the runtook 3m 25s
startup 52.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 52.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 31.6 s (87,384 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed4h 53m into the runtook 3m 53s
startup 108.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 108.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.4 s (221,580 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 3passed6h 27m into the runtook 4m 47s
startup 133.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 133.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 23.9 s (115,401 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 4passed6h 55m into the runtook 4m 18s
startup 122.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 122.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 22.0 s (125,351 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is over. The final output passed its last H100 self-check: 7.32 docs/s against the 5 required, and /health answered in 122.7 s of the 300 s allowed.
The model is four fine-tuned checkpoints averaged together: three ModernBERT-large and one DeBERTa-v3-large, each with a character-level head. On the 5-fold cross-validation over the 39 labelled documents, the three ModernBERT checkpoints alone scored 0.7612 and adding DeBERTa gave 0.7705. That gain is small next to the 0.01–0.02 noise in the cross-validation.
DeBERTa is slower than the ModernBERT members, so the server drops it under heavy load. While at least 6 requests are in flight, it skips DeBERTa if the service rate over the last 24 requests falls below 6 docs/s, and brings it back above 7.5 docs/s. In the final self-check this happened twice, so some documents were averaged over the three ModernBERT checkpoints only. Without this, the four-checkpoint version measured 6.74 docs/s, which leaves too little margin if the held-out documents turn out longer than the public ones. architecture.json describes this behaviour.
/workspace/output holds serve.sh, app/, model/ and model_deb/, the vendored wheels, train/, logs/, architecture.json and DATA_MANIFEST.md.