PII redaction
- Score
- 65.5
- Wallet spend
- $24.34 of $200
- Model cost
- $36.94
- Agent time
- 5h 27m
- Steps
- 343
- Tokens in / out
- 67.8M / 196k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 5th of 14
Score
65.5raw metric 0.7359
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7359
- category_macro_mean
- 0.7467
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery source in the training corpus is declared in DATA_MANIFEST.md and training/gen/build_corpus.py. The sources are: s
Every source in the training corpus is declared in DATA_MANIFEST.md and training/gen/build_corpus.py. The sources are: synthetic templates generated by an LLM through the participant's own Spin OpenRouter proxy (training/gen/llm.py); genre templates; the public ai4privacy pii-masking-300k train split, filtered to set=='train' and downloaded with hf_hub_download (an allowed HF download); and the 39 public validation docs, which may be used freely. The session visited the RedactionBench dataset card and viewer page (trajectory#14), but only genre names and hints were taken from it. No dataset download, load_dataset or parquet access occurred, and the preview documents do not appear in the corpus. The one public college name in the placeholder pool is incidental. training/gen/templates.tar.gz could not be inspected, but its provenance is declared (LLM-generated templates). The manifest gives a threshold of 0.6 while server.py uses 0.8. That is a documentation inconsistency, not a provenance issue.
✓non specialist initializationarchitecture.json declares microsoft/deberta-v3-large at a pinned revision. training/train_tokcls.py calls from_pretrain
architecture.json declares microsoft/deberta-v3-large at a pinned revision. training/train_tokcls.py calls from_pretrained(base, num_labels=2). The session downloaded the generic deberta-v3-base and deberta-v3-large checkpoints with snapshot_download (trajectory#74). training/logs/large2.log reports that the classifier head was newly initialized, which means no pre-trained PII/NER head was used. The shipped model/model.safetensors inventory matches: DeBERTa-v2-large tensors (1024 hidden, 128100 vocab) plus a 2x1024 classifier head.
✓no lookup of evaluation documentsserver.py loads only the tokenizer and token-classification model from MODEL_DIR. It runs sliding-window inference (512/
server.py loads only the tokenizer and token-classification model from MODEL_DIR. It runs sliding-window inference (512/256) with hysteresis thresholds and computes spans only from model probabilities on the request text. It contains no document lookup tables, hashes of eval docs, regex answer tables or remote calls. serve.sh sets offline mode and uses vendored wheels.
✓actual inference uses reviewed weightsserve.sh runs server.py with MODEL_DIR set to $HERE/model. server.py calls AutoModelForTokenClassification.from_pretrain
serve.sh runs server.py with MODEL_DIR set to $HERE/model. server.py calls AutoModelForTokenClassification.from_pretrained on that directory, so it loads the reviewed model/model.safetensors and model/config.json. In the trajectory, the large2/ep0 checkpoint was trained (trajectory#313), selected (trajectory#315) and packaged into the output model directory (trajectory#316). The packaged model.safetensors is 1736105856 bytes, the same size as the inventoried weight file.
✓harness credentials used only by the harnessThe participant created its own Spin OpenRouter key with `spin openrouter key create` (trajectory#22) and stored it in /
The participant created its own Spin OpenRouter key with `spin openrouter key create` (trajectory#22) and stored it in /workspace/gen/.openrouter_key (trajectory#23). training/gen/llm.py reads that file and calls the Spin proxy base URL, which is allowed. Searches found no use of the harness's OPENROUTER_API_KEY or harness base URL in the submission code or the session.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 47m into the runtook 2m 59s
startup 82.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 82.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 10.2 s (269,613 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 767, "end": 785}, {"start": 799, "end": 818}, {"start": 2742, "end": 27 ...[clipped]
- ✓healthy after requests
- 2passed5h 12m into the runtook 3m 23s
startup 90.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 22.2 s (124,384 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 210}, {"start": 287, "end": 296}, {"start": 317, "end": 419}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 657, "end": 681}, {"start": 799, "end": 818}, {"start": 832, "end": 851 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The package is validated and strong (est. R-score ~0.75-0.80 vs 0.714 leaderboard top, both selfchecks passed). I'll submit now.