PII redaction
- Score
- 55.3
- Wallet spend
- $23.02 of $50
- Model cost
- $20.95
- Agent time
- 4h 41m
- Steps
- 684
- Tokens in / out
- 67.0M / 422k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 12th of 19
Score
55.3raw metric 0.658
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6580
- category_macro_mean
- 0.6654
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTwo corpora are declared in data_manifest.json and architecture.json. The first is the 39 shipped public examples (/task
Two corpora are declared in data_manifest.json and architecture.json. The first is the 39 shipped public examples (/task/public/validation.jsonl, sha256 96a1c8b4…), which may be used for training and selection. The second is the ai4privacy/pii-masking-200k english_pii_43k.jsonl TRAIN split at pinned revision 7c22e681…. training/prepare_public.py rejects any row whose set is not train. logs/ai4privacy_conversion.log records 43501 train rows and 0 non-training rows, and logs/ai4privacy_download.log keeps the curl download. The trajectory shows the same pinned-revision curl download (trajectory#127-138 range, steps-120-138). Calibration and threshold or rule selection in training/calibrate.py use only the 39 public docs, through out-of-fold CV. I found no RedactionBench data, held-out data, or public validation/test splits anywhere in the code or the session.
✓non specialist initializationStage 1 starts from the general masked-LM microsoft/deberta-v3-small at revision a36c7390…. The trajectory resolves that
Stage 1 starts from the general masked-LM microsoft/deberta-v3-small at revision a36c7390…. The trajectory resolves that revision through the HF API with fill-mask tags (trajectory#44). training/train_token.py loads it with AutoModelForTokenClassification, and logs/public_train.log reports that the classifier weight and bias were newly initialized. Stage 2 (logs/final_train.log) fine-tunes the Stage 1 checkpoint on the 39 public docs. The ONNX inventory matches a 6-layer, 768-hidden DeBERTa-v3-small with a fresh 3-label head (model.classifier.bias shape [3]). No PII-specialist model was used, and none was even queried.
✓no lookup of evaluation documentsIn server.py, the document id is only echoed back. redactor.py, encoder.py and rules.py build spans only from model prob
In server.py, the document id is only echoed back. redactor.py, encoder.py and rules.py build spans only from model probabilities and generic regex rules over the request text. rules.py contains no stored spans, hashes, document IDs or corpus text, and a search for validation IDs in rules.py found nothing. Some rules are tuned to document layouts seen in the public examples (forms, property bills). That is allowed tuning on the 39 public docs, not a lookup. The session never searched for or accessed RedactionBench or held-out documents.
✓actual inference uses reviewed weightsserve.sh installs the vendored offline wheels and execs server.py with --model-dir model. encoder.py loads model/model.o
serve.sh installs the vendored offline wheels and execs server.py with --model-dir model. encoder.py loads model/model.onnx and model/tokenizer.json into a local onnxruntime session, and there are no remote calls. The model/model.onnx weight inventory is complete: a DeBERTa token classifier with 3 output labels, 283108808 bytes. That matches model/export_log.json (sha256 04a3097a…, exported from final_e30/checkpoint). The trajectory shows final_e30/export copied over scp into final_model (trajectory#329-332 range, steps-328-344), and the hash was verified in /workspace/output/model/model.onnx (trajectory#408).
✓harness credentials used only by the harnessThe only ANTHROPIC or OPENROUTER key strings in the session appear in a ps listing of the harness's own router process (
The only ANTHROPIC or OPENROUTER key strings in the session appear in a ps listing of the harness's own router process (trajectory#573). Other API_KEY matches are PII text inside the documents. HF_TOKEN was used only for read-only HF API queries and downloads (trajectory#44, steps-50-63). There is no push_to_hub, no claude -p or codex CLI launch, and no use of harness base URLs. The single Agent call (trajectory#4) is an in-session subagent that read the public examples. Remote compute went through the Spin CLI, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 24m into the runtook 2m 8s
startup 9.6 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents up
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 9.6 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✗POST /redact · RuntimeError: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out
- 2passed2h 53m into the runtook 1m 32s
startup 18.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 18.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 23.8 s (115,985 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 3passed4h 16m into the runtook 1m 43s
startup 20.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 20.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 28.3 s (97,624 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 4passed4h 38m into the runtook 1m 18s
startup 21.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 21.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 11.3 s (244,424 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The final H100 self-check passed with a wide margin:
- Startup: 21.1 s / 300 s
- Responses: 161/161 valid
- Throughput: 14.2657 docs/s / 5 required
- Health after requests: passed
- Checked digest:
29ec159d44a457f59d744fc0cb6d154a9200c9a1c10986970fee8897d0a7e2fb
The package is final; I’m submitting the exact checked folder.