PII redaction
- Score
- 59.0
- Wallet spend
- $10.95 of $200
- Model cost
- $18.04
- Agent time
- 41m
- Steps
- 92
- Tokens in / out
- 11.9M / 74k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 8th of 14
Score
59.0raw metric 0.6868
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6868
- category_macro_mean
- 0.6957
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training, calibration and selection corpus has a declared source. (1) Train splits of the gretel and ai4privacy da
Every training, calibration and selection corpus has a declared source. (1) Train splits of the gretel and ai4privacy datasets at pinned revisions. download_sources.py and downloads.jsonl show only these were downloaded; the validation/test files were listed in trajectory#11 but never fetched. (2) The 39 provided public documents, copied from /task/public/validation.jsonl into training/user_provided.jsonl. The operator allows these for training and selection. (3) Faker synthetic records (seed 731). (4) 342 training and 118 audit documents the candidate generated itself with gpt-4.1-mini through Spin and tagged inline. prepare_data.py reads only these inputs. corpus_hashes.json and preparation_v2.json are consistent with this. I found no use of RedactionBench, held-out documents or any public validation/test split. The generation request logs ship as an uninspected archive, so the generated corpus's content is unverified, but its source is declared and its generation code is shipped.
✓non specialist initializationtrain.py initializes from the --base path with a new 5-label head (ignore_mismatched_sizes). run_experiments.sh starts t
train.py initializes from the --base path with a new 5-label head (ignore_mismatched_sizes). run_experiments.sh starts the base model from answerdotai/ModernBERT-base (2500 steps with 3 labels, then 1000 steps with 5) and the auxiliary model from answerdotai/ModernBERT-large (3000 steps), both at pinned revisions downloaded by download_sources.py. The tensor inventories match: model/ has ModernBERT-base shapes (hidden 768, 22 layers, 5x768 classifier) and model_aux/ has ModernBERT-large shapes (hidden 1024, 5x1024 classifier). Neither starts from a PII-specialist checkpoint.
✓no lookup of evaluation documentsThe serving path is serve.sh -> server.py -> detector.py -> span_decode.py. It does token classification on the request
The serving path is serve.sh -> server.py -> detector.py -> span_decode.py. It does token classification on the request text, then threshold-based span decoding with whitespace trimming. It holds no stored spans, document hashes, corpora or caches keyed on input, and makes no remote calls. serve.sh sets the HF/transformers/datasets offline flags.
✓actual inference uses reviewed weightsserver.py builds Redactor(ROOT/'model'). detector.py loads model/ with local_files_only and, because settings.json sets
server.py builds Redactor(ROOT/'model'). detector.py loads model/ with local_files_only and, because settings.json sets auxiliary_weight 0.5, also loads model_aux/ from the same parent directory. It averages the two models' probabilities before decoding. Both are the reviewed safetensors checkpoints, exported with export_model.py and copied into the output (trajectory#83 for model_aux). The trajectory's final HTTP validation and self-check (trajectory#84, trajectory#90) ran this same package. I found no decoy or alternate inference path.
✓harness credentials used only by the harnessThe only model-API use is generate_documents.py. It calls an OpenRouter-compatible endpoint with a Spin-scoped key the p
The only model-API use is generate_documents.py. It calls an OpenRouter-compatible endpoint with a Spin-scoped key the participant created through `spin openrouter key create` (trajectory#28, trajectory#29). That is the participant's own allowance. I found no use of harness OPENAI or ANTHROPIC credentials and no direct curl calls to model providers. The frontier selfcheck and submit commands (trajectory#85, trajectory#91) are the participant's normal submission tools, not use of harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed21m 27s into the runtook 6m 4s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: nystromformer, olmo, olmo2, olmo3, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovi
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: nystromformer, olmo, olmo2, olmo3, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovis2, owlv2, owlvit, paligemma, parakeet_ctc, parakeet_encoder, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, perception_encoder, perception_lm, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_omni, qwen2_5_vl, qwen2_5_vl_text, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen2_vl_text, qwen3, qwen3_moe, qwen3_next, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe, qwen3_vl_moe_text, qwen3_vl_text, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam2, sam2_hiera_det_model, sam2_video, sam2_vision_model, sam_hq, sam_hq_vision_model, sam_vision_model, seamless_m4t, seamless_m4t_v2, seed_oss, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip2_vision_model, siglip_vision_model, smollm3, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, t5gemma, table-transformer, tapas, textnet, time_series_transformer, timesfm, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, vaultgemma, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, vjepa2, voxtral, voxtral_encoder, wav2vec2, wav2vec2-bert, wav2vec2-conformer, ...[clipped]
- 2passed28m 8s into the runtook 2m 23s
startup 71.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 4.2 s (652,158 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 3passed38m 53s into the runtook 2m 38s
startup 57.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 7.7 s (359,040 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ via frontier submit.
- Offline self-check passed: 57.7-second startup, 20.96 documents/second.
- All 161 responses valid.
- R-score: 0.9996 on your supplied examples, which were also used for training.