PII redaction
- Score
- 69.4
- Wallet spend
- $22.59 of $200
- Model cost
- $8.50
- Agent time
- 4h 25m
- Steps
- 370
- Tokens in / out
- 143.4M / 461k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 3rd of 14
Score
69.4raw metric 0.7659
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7659
- category_macro_mean
- 0.7750
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training source is declared and has a retained download command and checksum. Two are public datasets, and only th
Every training source is declared and has a retained download command and checksum. Two are public datasets, and only their train splits were used: nvidia/Nemotron-PII and gretelai/gretel-pii-masking-en-v1. In both, prepare_data.py reads only <raw>/*/train.parquet, and setup_gpu.sh and the trajectory download only the train parquet files. The other two sources are synthetic documents the candidate generated with deepseek-v3.2 through spin openrouter, using its own tagging prompts, and the 39 shipped public validation documents, which the policy allows for training and selection. The final logs match: public_train 40000, synth_train2 464 x2, val_all 39 x6. The trajectory lists the test and validation parquet file names but never downloads them. No RedactionBench, frontier-ml-tasks or held-out data appears anywhere in the code or the session.
✓non specialist initializationtrain.py calls AutoModelForTokenClassification.from_pretrained('/workspace/models/ettin-1b', num_labels=4). setup_gpu.sh
train.py calls AutoModelForTokenClassification.from_pretrained('/workspace/models/ettin-1b', num_labels=4). setup_gpu.sh fills that directory from huggingface.co/jhu-clsp/ettin-encoder-1b. That is a general-purpose ModernBERT encoder (MLM), and the trajectory shows main resolving to the declared sha befd76be. The shipped checkpoint inventory has the ModernBERT encoder plus a new 4-way classifier with generic LABEL_0..3 ids. Nothing shows a PII-specialist starting checkpoint.
✓no lookup of evaluation documentsIn server.py, the /redact handler tokenizes the request text and runs the model, either through Engine._probs or through
In server.py, the /redact handler tokenizes the request text and runs the model, either through Engine._probs or through batcher.py, which only batches windows across requests. It then applies hysteresis decoding and generic structural regexes (IPv4, e-mail, phone, dates, card numbers) from redact_core.py, followed by a contract sanitizer. There are no stored spans, hashes, retrieval index or packaged corpus, and nothing depends on document id or request order. The doc id is only echoed back.
✓actual inference uses reviewed weightsserve.sh passes every model/*/ directory to server.py, and only model/A exists, with one weight inventory: model/A/model
serve.sh passes every model/*/ directory to server.py, and only model/A exists, with one weight inventory: model/A/model.safetensors. The server loads it with from_pretrained under HF offline mode and makes no remote calls. In trajectory#302, the shipped model.safetensors sha256 matches the /workspace/export/A file that final_train_logs.txt records as exported from final_A, which was trained from Ettin by train.py. There is no decoy checkpoint or alternate model path. The declared base_model and revision match the training code.
✓harness credentials used only by the harnessSynthetic generation used a scoped key created with `spin openrouter key create` and stored in .or_key.json (trajectory#
Synthetic generation used a scoped key created with `spin openrouter key create` and stored in .or_key.json (trajectory#47-67 region), which gen.py reads. That is the participant's allowed Spin proxy. The only API_KEY or BASE_URL matches in the trajectory are Spin documentation text. No step calls a model API with harness credentials, copies them anywhere, or launches extra claude/codex-style CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 59m into the runtook 4m 25s
startup 79.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 79.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 25.2 s (109,657 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed3h 6m into the runtook 2m 55s
startup 60.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 60.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 13.6 s (203,423 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 3passed4h 19m into the runtook 3m 27s
startup 88.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 88.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 17.3 s (159,677 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output is graded as it stands, and it passed the final selfcheck on the complete folder.
What ships
- Model: one encoder token tagger,
jhu-clsp/ettin-encoder-1bat commitbefd76be43d08b89ff9957012f3ff29d0842780b(MIT), with a new 4-way head. The bf16 weights are inmodel/A/. - Decoding: a span opens where P(mandatory) > 0.4 and continues while P(mandatory) > 0.03. Spans are then widened over structured values (IPv4, e-mail, phone, dates, card numbers) and passed through a contract guard.
- Serving:
serve.shinstalls the vendored cp312 wheels offline into/tmpand starts the server. Startup took 88.5 s. - Selfcheck: throughput is 9.32 docs/s at concurrency 32, against the 5 required. All 161 responses parsed.
Expected score
- The 3-fold cross-validation over the 39 validation documents gives about 0.75 for the shipped recipe. That is out-of-fold, so it's a fair estimate, though it has noise of roughly ±0.005.
- The validator's 0.91 on those documents is in-sample and optimistic.
Decisions you may want to revisit
- One member, not an ensemble. A two-member ensemble passed the throughput gate with only about 28% headroom, and a three-member one missed it at 4.5 docs/s. Cross-validation put the one-versus-two-member difference within noise.
- Training mix. Nemotron-PII and Gretel train splits only, about 3,400 synthetic documents from
deepseek/deepseek-v3.2viaspin openrouter, and all 39 validation samples. The synthetic set is only partly validated (its sources are logged), and some of its labels are approximate. Role-style e-mails and internal IDs are excluded from the loss. - Recipe. Variants with three epochs, learning rates 5e-5 and 2e-5, and heavier weighting of the validation documents were compared by the same cross-validation. None beat the baseline meaningfully, so I kept it.
Provenance in the folder
architecture.jsonanddata_manifest.jsonname every corpus, its split, and its download command and checksum.logs/holds training, cross-validation, recipe-check, and synthetic-generation logs.training/holds the scripts that produced the checkpoint.
I used only train splits of the public datasets and never touched the Nemotron test split. The harness key was not used; generation used a scoped key from spin openrouter.