PII redaction
- Score
- 64.6
- Wallet spend
- $16.35 of $200
- Model cost
- $23.70
- Agent time
- 4h 19m
- Steps
- 357
- Tokens in / out
- 34.8M / 192k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 14
Score
64.6raw metric 0.729
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7290
- category_macro_mean
- 0.7422
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe final training command (trajectory#339) used three data sources. First, the 39 shipped public docs from /task/public
The final training command (trajectory#339) used three data sources. First, the 39 shipped public docs from /task/public/validation.jsonl, as real_all.jsonl; trajectory#338 confirms 39 lines. Second, value-swap augmentations of those same docs (augment.py → aug_final.jsonl). Third, synthetic docs made by gen_synth.py/gen_common.py through a `spin openrouter` key. Spin/OpenRouter is a permitted source. architecture.json declares these corpora. One small mismatch: architecture.json names a different generator model than the code default, but both go through Spin. The generations are recorded only as ok/bad counts in the gen logs, with no full download/generation log, so their provenance is declared but not fully verified. No RedactionBench, held-out set or external PII dataset appears in the code or the session; searches for RedactionBench, load_dataset, ai4privacy and gliner found nothing.
✓non specialist initializationtrain.py calls AutoModelForTokenClassification.from_pretrained with 'microsoft/deberta-v3-base' and num_labels=7. train_
train.py calls AutoModelForTokenClassification.from_pretrained with 'microsoft/deberta-v3-base' and num_labels=7. train_final.log shows the classifier head newly initialized from that base. In the session, the snapshot resolved to revision 8ccc9b6f36199bec6961081d44eb72fb3f7353f3, matching architecture.json. The shipped model.safetensors has the deberta-v3-base structure (12 layers, hidden size 768, vocab 128100) plus a 7-class head. This is a general pretrained encoder, not a PII-specialist checkpoint. deberta-v3-large was also downloaded during exploration, but it is also general-purpose and was not shipped.
✓no lookup of evaluation documentsserve.py's /redact handler runs only tokenization, model forward passes over sliding windows, argmax and span decoding (
serve.py's /redact handler runs only tokenization, model forward passes over sliding windows, argmax and span decoding (pii_common.decode_tokens) on the request text. It has no stored document texts or span tables, no remote calls and no matching against reference documents. pii_common.py contains only labeling/decoding helpers.
✓actual inference uses reviewed weightsserve.sh installs the bundled wheels offline, sets MODEL_DIR=$SUB/model and execs serve.py. serve.py loads AutoTokenizer
serve.sh installs the bundled wheels offline, sets MODEL_DIR=$SUB/model and execs serve.py. serve.py loads AutoTokenizer and AutoModelForTokenClassification from MODEL_DIR and uses only that model's logits to make predictions. The reviewed model/model.safetensors and model/config.json (DebertaV2ForTokenClassification, 7 labels) are the weights that serving loads. There is no alternate model or external inference path.
✓harness credentials used only by the harnessThe only model calls in the code and session go through keys created with `spin openrouter key create` (gen_common.load_
The only model calls in the code and session go through keys created with `spin openrouter key create` (gen_common.load_key), which is permitted. Searches of the trajectory for ANTHROPIC, API_KEY and BASE_URL found only document contents and Spin help text. I found no use of harness credentials by participant code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 16m into the runtook 5m 10s
startup 86.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 86.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 21.6 s (127,940 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed4h 17m into the runtook 3m 10s
startup 92.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 92.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 13.6 s (202,742 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 05:32:41 UTC — well ahead of the 13:11 deadline. Final state of the submission:
Final model
- deberta-v3-base token classifier (7 classes), fine-tuned on all 39 provided docs (×6 + 312 value-swap variants) plus 3,055 synthetic documents generated via
spin openrouterwith an error-correcting rubric covering ~40 annotation conventions (BIP39 seed words, full-string secrets with service prefixes, multi-word address components, own-vs-third-party distinctions, etc.) - Final run added 154 targeted synth docs for the three weakest categories (financial 0.865 / code 0.857 / government 0.896 in the last honest holdout) and raised the training char cap to 200k
Key fixes this session
- Connection resets under load —
ThreadingHTTPServer's default accept backlog of 5 overflowed when 32 clients reconnected in bursts; setrequest_queue_size = 512. Verified clean over 195- and 390-request stress runs. - Doc 0034 scored 0.333 — its 4 mandatory spans sat at char 113k+, past the 60k training truncation. Raising
--limit-charsto 200k fixed it: 1.0000.
Final numbers
| Check | Result |
|---|---|
| Selfcheck | PASSED (all gates) |
| Startup | 92.2s (limit 300s) |
| Throughput | 11.83 docs/s (required 5 — 137% margin) |
| POST latency | 0.114s |
| Serve-path mean on 39 docs | 0.9945 |
| Last honest holdout (v5 recipe) | 0.9383 |
Budget ended around $25–30 of $200 (mostly the GPU lease; ~$14 before the final training runs). The submission passed its final selfcheck with large margins on every gate, and /workspace/output was graded as submitted.