PII redaction
- Score
- 64.3
- Wallet spend
- $4.61 of $10
- Model cost
- $6.37
- Agent time
- 3h 4m
- Steps
- 294
- Tokens in / out
- 104.2M / 405k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 6th of 14
Score
64.3raw metric 0.7269
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7269
- category_macro_mean
- 0.7317
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata/manifest.json declares every corpus. The 39 shipped public validation documents may be used for any purpose. The fo
data/manifest.json declares every corpus. The 39 shipped public validation documents may be used for any purpose. The four synthetic runs (119, 249, 375 and 86 hard-negative documents) were generated and annotated by deepseek-v3.2 through the Spin OpenRouter proxy. The generation code (data/synthetic/gen.py), the generation logs and the per-run document files are all retained. train/COMMANDS.md and logs/gpu/final_v2.log show the final model trained only on validation.jsonl (repeated 3x) plus syn1234.jsonl: 946 documents in total, 829 of them synthetic. The training code, data files and session record contain no RedactionBench data beyond the public examples, no use of held-out documents, and no load_dataset calls or HF dataset downloads, including any validation or test splits.
✓non specialist initializationarchitecture.json and train/train_cv.py both use answerdotai/ModernBERT-large, a general-purpose encoder, at revision 45
architecture.json and train/train_cv.py both use answerdotai/ModernBERT-large, a general-purpose encoder, at revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13 via snapshot_download. logs/gpu/final_v2.log records that snapshot, and it records that the classifier head was newly initialized. The checkpoint matches this. model/config.json declares ModernBertForTokenClassification with 3 generic labels (LABEL_0..2). The tensors in model/model.safetensors are standard ModernBERT tensors plus a head and a 3x1024 classifier. An ettin-encoder-1b run was tried during CV but abandoned; ettin is also a general-purpose encoder. No PII-specialist checkpoint was used.
✓no lookup of evaluation documentsserve.sh runs code/server.py offline (HF_HUB_OFFLINE=1). On each request, the /redact handler tokenizes the input text,
serve.sh runs code/server.py offline (HF_HUB_OFFLINE=1). On each request, the /redact handler tokenizes the input text, runs windowed model inference (code/model_io.py predict_token_probs), and decodes spans by thresholding P(mandatory) at 0.40 (code/redact_core.py decode_spans). The serving path contains no stored spans, document hashes, retrieval tables or remote calls. code/block_attn.py is only an attention speed optimization.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR to the shipped model folder. code/server.py loads model/decision.json and model/tokenizer.json,
serve.sh sets MODEL_DIR to the shipped model folder. code/server.py loads model/decision.json and model/tokenizer.json, and loads the weights via model_io.load_model, which calls AutoModelForTokenClassification.from_pretrained on that folder. That folder contains model/model.safetensors, the reviewed checkpoint in the weight inventory. Every response comes from this model's token probabilities. The trajectory shows the final v2 weights copied into the output model folder with a successful cmp check, and a later re-check also printed 'weights ok'. All wheels installed by serve.sh match approved upstream digests.
✓harness credentials used only by the harnessThe participant created their own scoped OpenRouter key with `spin openrouter key create` (trajectory step 58). That key
The participant created their own scoped OpenRouter key with `spin openrouter key create` (trajectory step 58). That key was written to .or_key.json and is used by data/synthetic/gen.py to call the Spin proxy, which the policy allows. HF_TOKEN was used only to query and download public models, which is also allowed. The session shows no reads or uses of harness-only credentials (e.g. no ANTHROPIC keys or environment dumps), and the shipped serving code uses no credentials at all.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 1m into the runtook 2m 27s
startup 65.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 9.3 s (297,382 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed2h 32m into the runtook 3m 21s
startup 80.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 80.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 19.8 s (139,427 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 3passed2h 45m into the runtook 3m 14s
startup 75.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 23.0 s (120,017 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 667, "end": 668}, {"start": 669, "end": 670}, {"start": 2742, "end": 2750}, {"start": 2751, "end": ...[clipped]
- ✓healthy after requests
- 4passed3h 0m into the runtook 2m 56s
startup 85.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 85.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.7 s (187,074 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 667, "end": 668}, {"start": 669, "end": 670}, {"start": 2742, "end": 2750}, {"start": 2751, "end": ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output is graded as it stands. It passed all four frontier selfchecks, and the last one ran on an H100 with 10.9 documents/s against a required 5.
What the server does
serve.shinstalls the vendored wheels into/tmp, then starts a stdlib HTTP server withPOST /redactandGET /health.- The model is ModernBERT-large (
answerdotai/ModernBERT-largeat45bb4654a4d5aaff24dd11d4781fa46d39bf8c13), fine-tuned as a 3-way token classifier. - It predicts mandatory tokens with P ≥ 0.40 over 8190-token windows, trims whitespace, and returns exact character spans.
- The sliding-window attention layers use an exact block-local computation. I checked it against the stock path: identical probabilities and spans on real documents, and identical outputs across the edge window lengths.
Quality
- Four-fold cross-validation over the 39 validation documents, scored with
spans.py, gives about 0.75 out-of-fold. The threshold plateau is flat from 0.30 to 0.75. - The final model is trained on all 39 documents, so its in-sample score (0.93) is not an estimate.
- A zero-shot teacher, prompted with the same policy and no training, scored 0.67 on mandatory spans. The student beats it.
Training data
- The 39 validation documents, which you allowed me to use.
- 829 synthetic documents generated by
deepseek/deepseek-v3.2through spin's OpenRouter proxy, with inline annotations. The guide, scenarios, generator, raw responses, and parsed documents are indata/synthetic/. The sources and sha256 checksums are indata/manifest.json. - No public benchmark split was used.
Decisions I made that you may want to revisit
- Contextual spans are ignored in the loss and never predicted. Predicting them caused partial-coverage penalties that dropped the oracle to 0.79.
- Context length is 8190 tokens rather than 2046, which added about 0.03 out-of-fold. It is also the main throughput cost. The selfcheck margin is 118%, which I judged safe.
- The threshold of 0.40 was set by hand from the flat part of the sweep, after the final training run.
train/COMMANDS.mdrecords that edit. - Ettin-1B was tested on one fold, scored lower, and was dropped.
Budget
- About $4.6 of the $10 wallet was spent on GPU leases and synthetic generation. The leases are released.