PII redaction
- Score
- 70.0
- Wallet spend
- $34.56 of $50
- Model cost
- $34.12
- Agent time
- 2h 19m
- Steps
- 138
- Tokens in / out
- 29.2M / 176k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 3rd of 14
Score
70.0raw metric 0.7709
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7709
- category_macro_mean
- 0.7798
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is synthetic. It was generated during the session through the Spin OpenRouter proxy, which the polic
The training corpus is synthetic. It was generated during the session through the Spin OpenRouter proxy, which the policy allows, using the code in data_generation/ (llm.py, bulk.py, prompt.py, convert.py). Its source is declared in architecture.json and DATA_MANIFEST.md, which give the generator models, commands and endpoint. The corpus is shipped as training_data/*.jsonl and training_data_extra/*.nolog.jsonl, and prepare.py and train.py consume only these files. The 39 public validation examples appear only as few-shot prompt material and in model selection, which is allowed. I found no load_dataset call, no download of a hidden or test split and no external PII corpus in the code or the trajectory.
✓non specialist initializationBoth models start from the general-purpose base microsoft/deberta-v3-large at revision 64a8c8ea..., as declared in archi
Both models start from the general-purpose base microsoft/deberta-v3-large at revision 64a8c8ea..., as declared in architecture.json. The trajectory shows the base being fetched with snapshot_download (trajectory#46, recorded in logs/download_log.txt), and every training run passes --model /workspace/models/deberta-v3-large to train.py, which calls from_pretrained with num_labels=3 (trajectory#49). The configs and safetensors inventories show a standard DebertaV2ForTokenClassification with generic labels LABEL_0/1/2 and a fresh classifier head of shape [3,1024]. Nothing shows initialization from a PII-specialized checkpoint.
✓no lookup of evaluation documentsServing is pure model inference. serve.py's /redact handler calls Redactor.predict from infer.py, which runs sliding-win
Serving is pure model inference. serve.py's /redact handler calls Redactor.predict from infer.py, which runs sliding-window token classification, applies the spans_from_probs_v3 thresholds plus generic regexes, and propagates repeated spans within the same document. No stored document corpus, hash table, retrieval index or network fetch of evaluation documents exists in the serving path.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, and runs serve.py. serv
serve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, and runs serve.py. serve.py reads decision.json, whose models list is [model, model2], and loads both local directories with AutoModelForTokenClassification.from_pretrained. It averages their probabilities and falls back to model/ alone only under load. The inventoried files model/model.safetensors and model2/model.safetensors are the weights actually served; the trajectory shows the serve.py ensemble edit and the m2/m2c bf16 exports (trajectory/steps-124-138). REDACT_MODEL is a generic local-path override that the harness does not set. The server makes no remote inference calls.
✓harness credentials used only by the harnessFor model calls during the session, the participant created a scoped Spin OpenRouter key with `spin openrouter key creat
For model calls during the session, the participant created a scoped Spin OpenRouter key with `spin openrouter key create` (trajectory#19); data_generation/llm.py reads that key from or_key.json and uses it for generation (trajectory#47, trajectory#48). Neither the code nor the trajectory shows any use of the harness's ANTHROPIC credentials or base URL, or any claude CLI subprocess workers.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 51m into the runtook 2m 59s
startup 76.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 76.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.0 s (172,843 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 2passed2h 15m into the runtook 3m 21s
startup 84.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 19.4 s (142,080 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Budget is $50 and the API model list is long. Before designing, I need to understand the annotation conventions across categories. Let me dump annotated views of several documents.