PII redaction
- Score
- 59.6
- Wallet spend
- $49.25 of $50
- Model cost
- $2.02
- Agent time
- 8h 2m
- Steps
- 808
- Tokens in / out
- 79.5M / 779k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 7th of 19
Score
59.6raw metric 0.6912
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6912
- category_macro_mean
- 0.7043
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.md lists two training sources. The first is the 39 public validation documents (/task/public/validation.js
data_manifest.md lists two training sources. The first is the 39 public validation documents (/task/public/validation.jsonl): 27 used for training and 12 held out for selection. The second is LLM documents that the participant generated with claude-haiku-4.5 and gpt-4.1-mini through the Spin OpenRouter proxy. The generator scripts (train/gen_llm.py, train/gen_llm_tmpl.py, train/llm_common.py) read only the public GOLD file. The generation logs are retained and the generated data/llm_*.jsonl files are shipped. The code and trajectory contain no RedactionBench, load_dataset, snapshot_download or other hidden-corpus download. Every source is declared and allowed.
✓non specialist initializationtrain/train_pt2.py calls AutoModelForTokenClassification.from_pretrained('/workspace/pt/roberta-base', num_labels=3). lo
train/train_pt2.py calls AutoModelForTokenClassification.from_pretrained('/workspace/pt/roberta-base', num_labels=3). logs/pt_dl.log shows FacebookAI/roberta-base being loaded: lm_head is UNEXPECTED and the classifier is MISSING, so the classification head starts fresh. That is a general-purpose encoder, not a PII-specialist checkpoint. model_roberta/config.json has generic labels LABEL_0 through LABEL_2 and the standard roberta-base shapes. The model.safetensors inventory contains roberta.* tensors plus a [3,768] classifier.
✓no lookup of evaluation documentsIn server.py, POST /redact queues the request text and runs BACKEND.predict(texts). torch_backend.py does windowed token
In server.py, POST /redact queues the request text and runs BACKEND.predict(texts). torch_backend.py does windowed token classification, averages probabilities per word, then applies the seed/grow threshold rule, drop_newlines and trim_ws. The numpy fallback (numpy_backend.py, infer.py) runs a CNN over hashed features. Nothing is keyed on the document id, and there is no memorized table or external retrieval.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline (their trusted binary audits are approved), sets MODEL_DIR=$DIR/model_robe
serve.sh installs the vendored wheels offline (their trusted binary audits are approved), sets MODEL_DIR=$DIR/model_roberta and runs server.py. server.py builds a TorchBackend from model_roberta (config.json and model.safetensors, which has a reviewed inventory). If that fails it falls back to NumpyBackend on model.npz, which also has a reviewed inventory. Neither serving path makes a remote model or API call.
✓harness credentials used only by the harnessThe participant's only credential use was a Spin OpenRouter key they created with `spin openrouter key create` and used
The participant's only credential use was a Spin OpenRouter key they created with `spin openrouter key create` and used for data generation through the allowed proxy (SPIN_OPENROUTER_KEY / SPIN_BASE_URL in train/llm_common.py). The trajectory shows no use of the harness's ANTHROPIC or other harness credentials by participant code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed5m 42s into the runtook 2m 13sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed9m 23s into the runtook 43s
startup 1.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 1.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 3.1 s (899,983 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": []}
- ✓healthy after requests
- 3failed2h 18m into the runtook 1m 37s
startup 0.9 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents up
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 0.9 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 767, "end": 785}, {"start": 799, "end": 818}, {"start": 832, "end": 851 ...[clipped]
- ✓healthy after requests
- 4passed3h 5m into the runtook 2m 19s
startup 68.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 8.7 s (318,684 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 654, "end": 681}, {"start": 1101, "end": 1116}, {"start": 2709, "end": 2728}, {"start": 2742, "end" ...[clipped]
- ✓healthy after requests
- 5passed5h 20m into the runtook 2m 49s
startup 66.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 66.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 7.0 s (395,678 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 767, "end": 785}, ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final submission — /workspace/output/
| Model | FacebookAI/roberta-base @ e2da8e2f811d1448a5b465c236feacd80ffbac7b, fine-tuned as a word-level 3-class token classifier (other / mandatory / contextual), exported fp16, served on the H100 with vendored cp312 CUDA wheels; a numpy CNN backend ships as fallback |
| Training data | 540 re-rendered copies of 27 labelled user documents + 3930 LLM-written synthetic documents (scenario + template-transfer) — no public dataset or benchmark split used for training/calibration/selection |
| Decision rule | per-word p(mand)+p(ctx) averaged over BPE pieces and 512-token windows → seed at >0.98, absorb neighbouring runs >0.40, trim span whitespace, never redact a line break |
| Mean document R-score | 0.9732 on all 39 labelled documents, 0.9667 on the 12 never-trained-on holdout documents, 0.9732 measured end-to-end through the server (logs/validate_final.log) |
| Runtime | 66.1 s startup (selfcheck), 23.1 docs/s (selfcheck), 292k chars/s, GET /health → 200, POST /redact → {"id":…,"spans":[{"start":…,"end":…}]} |
What changed in this last stretch: the newline rule was the only post-processing win kept (+0.011 all-39, +0.010 holdout); contextual-edge trimming and a probability ensemble with the mix15 checkpoint were measured and rejected as harmful or not worth their cost, and all five corpus variations (mix8–mix15) were rejected because they regressed the holdout or the mean. Evidence for each rejection is in data_manifest.md §6 and the shipped logs, and architecture.json lists every training/inference file with all paths present in the submission.