PII redaction
- Score
- 56.1
- Wallet spend
- $31.41 of $50
- Model cost
- $23.14
- Agent time
- 4h 20m
- Steps
- 317
- Tokens in / out
- 61.6M / 198k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 8th of 14
Score
56.1raw metric 0.6644
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6644
- category_macro_mean
- 0.6744
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTwo sources make up the training corpus, and both are allowed. The first is the 39 shipped public validation documents p
Two sources make up the training corpus, and both are allowed. The first is the 39 shipped public validation documents plus 8 value-substituted augmentations of each (src/gen_data.py, src/augment.py, logs/gen_data.log). The second is 25,000 synthetic documents built by the candidate's own generators from hand-written word lists (src/gen_synth.py, src/synth.py, src/builder.py, src/names.py, logs/gen_synth.log). That adds up to 25,351 docs, which matches 'docs: 25351' in logs/train_final.log. The session confirms the concatenation (trajectory#188) and the final training run on it (trajectory#189). Threshold and checkpoint selection used only the shipped public examples (logs/eval_cv_sweep.log), which the operator allows. No dataset loader, RedactionBench reference, parquet file or held-out/task-repo access appears anywhere in the source or trajectory. One small inconsistency: data_manifest.json gives an older generator command (20000 docs, seed 12), while the retained log shows 25000 docs, seed 33. Both use the same declared local generators, so provenance is unaffected.
✓non specialist initializationarchitecture.json declares answerdotai/ModernBERT-base at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8, a general-p
architecture.json declares answerdotai/ModernBERT-base at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8, a general-purpose masked-LM encoder. The session resolved and downloaded that commit (trajectory#50). src/train.py builds CharPIIModel by calling AutoModel.from_pretrained('answerdotai/ModernBERT-base'), and logs/train_final.log shows the ModernBERT load report, with only the MLM head keys unexpected. The checkpoint inventory has ModernBERT encoder tensors plus a newly added char-conv head, and nothing that points to a PII-specialist checkpoint.
✓no lookup of evaluation documentsserve.sh runs app/server.py. Its /redact handler calls predict_probs, a sliding-window model forward pass over the text
serve.sh runs app/server.py. Its /redact handler calls predict_probs, a sliding-window model forward pass over the text of the current request, then probs_to_spans, which applies a threshold, hysteresis and generic filters for non-alphanumeric and short isolated spans. Finally it enforces the span contract. There are no stored spans, hashes, retrieval index, packaged corpus, id-keyed logic or request-order logic.
✓actual inference uses reviewed weightsapp/server.py loads app/weights.pt with torch.load and load_state_dict into CharPIIModel. Because app/ ships only config
app/server.py loads app/weights.pt with torch.load and load_state_dict into CharPIIModel. Because app/ ships only config.json and no base safetensors, the model is built with from_config, so every weight served comes from the reviewed checkpoint. The inventory's tensor names and shapes match app/model.py. The session shows app/weights.pt was copied from runs/final/model_s2100.pt, the output of the final training run (trajectory#189 and the copy steps in trajectory/steps-277-317). HF offline mode is set, and there are no remote calls, decoy checkpoints or alternate model paths.
✓harness credentials used only by the harnessThe trajectory shows no use of the harness's API keys or base URL from participant code. Searches for anthropic, openai,
The trajectory shows no use of the harness's API keys or base URL from participant code. Searches for anthropic, openai, OPENROUTER, BASE_URL and harness CLI launches (claude, kimi) found nothing. The only credential used was HF_TOKEN, for read-only downloads of the ModernBERT config and tokenizer (trajectory/steps-68-103), which is allowed. Compute ran through the Spin CLI.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 14m into the runtook 4m 27s
startup 77.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.0 s (172,459 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 656, "end": 681}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "en ...[clipped]
- ✓healthy after requests
- 2passed3h 57m into the runtook 2m 29s
startup 75.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 15.8 s (174,236 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 799, "end": 818}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end" ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything is verified and complete. Let me stop the GPU box (work done, saves budget) and submit: