PII redaction
- Score
- 74.8
- Wallet spend
- $7.47 of $10
- Model cost
- $44.32
- Agent time
- 3h 13m
- Steps
- 173
- Tokens in / out
- 37.6M / 203k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 1st of 14
Score
74.8raw metric 0.8071
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.8071
- category_macro_mean
- 0.8215
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining data has two declared sources: the 39 public task validation docs, which may be used, and synthetic documents.
Training data has two declared sources: the 39 public task validation docs, which may be used, and synthetic documents. training/generate.py makes the synthetic documents by calling general LLMs (deepseek-v4-flash, gpt-oss-120b, deepseek-v4-pro) through the Spin OpenRouter proxy. data_manifest/README.md and architecture.json declare both sources, and the README gives the generation commands. In the trajectory, the training tarball is built only from /task/public/validation.jsonl and gen/syn_*.jsonl (step 62). final_train_log shows 2137 synthetic docs and 39 GT docs. Nothing indicates RedactionBench data, hidden evaluation data or other undeclared corpora. The synthetic-data generation runs are only partly preserved through the logged session, which is acceptable.
✓non specialist initializationtraining/train.py initializes the encoder with AutoModel.from_pretrained('microsoft/deberta-v3-large'), a general-purpos
training/train.py initializes the encoder with AutoModel.from_pretrained('microsoft/deberta-v3-large'), a general-purpose encoder. remote_setup.sh caches it at the pinned revision 64a8c8ea..., matching architecture.json. The new content and whitespace heads start from random initialization. final_train_log confirms base microsoft/deberta-v3-large, and the trajectory shows runs/F/model.pt converted to bf16 and shipped as model/model.pt. Its inventory is DeBERTa-v3-large-shaped encoder.* tensors plus the heads. There is no PII-specialist or NER checkpoint.
✓no lookup of evaluation documentsThe /redact path in server.py runs tokenization, windowed model inference, aggregation, then decode_v3 and span validati
The /redact path in server.py runs tokenization, windowed model inference, aggregation, then decode_v3 and span validation. common.py's decode functions threshold the model probabilities and add a generic capitalized-name regex, applied only inside low-p_O regions the model has already flagged. No document corpus, stored spans, hashes, or id- or order-keyed behavior exists. The request id is only echoed back, and there are no network calls at serve time.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and points MODEL_DIR at the shipped model directory. server.py bui
serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and points MODEL_DIR at the shipped model directory. server.py builds DeBERTa from model/config.json and loads model/model.pt with torch.load and load_state_dict. That file's reviewed inventory is the complete bf16 DeBERTa-v3-large plus heads. The optional torch.compile wraps the same model, checks its output against eager, and falls back to eager on mismatch. compile_cache holds only Inductor/Triton JIT kernel artifacts, consumed through cache environment variables; the uninspected cuda_utils .so is a standard Triton helper, not an alternative model. No decoy or remote inference path exists.
✓harness credentials used only by the harnessThe trajectory shows the participant made its own OpenRouter key with `spin openrouter key create` (trajectory#22), and
The trajectory shows the participant made its own OpenRouter key with `spin openrouter key create` (trajectory#22), and generate.py uses it through /tmp/or_key.json. That is allowed participant use of Spin. Compute leases and S3 storage use the participant's own Spin session credentials. The only BASE_URL match is in the Spin docs text, and no harness API keys, Anthropic/OpenAI harness credentials or extra harness sessions are used.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 42m into the runtook 3m 23s
startup 90.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 25.8 s (106,911 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 2passed3h 2m into the runtook 3m
startup 97.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 97.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.7 s (216,526 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 3error3h 9m into the runtook 1m 59sfree (stopped before any GPU time)
The self-check could not finish (NotFoundError: Modal Sandbox with container ID ta-01M42B5DSQRE4BPEBXF94G7H6S not found. This means this Sandbox has already shut down. (Error code: 6SRG67PX)).
1 gates
- ✓review prerequisites
- 4passed3h 11m into the runtook 3m 22s
startup 102.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 102.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.9 s (214,664 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
All work is complete and verified. Submitting now.