PII redaction
- Score
- 52.5
- Wallet spend
- $23.02 of $200
- Model cost
- $39.09
- Agent time
- 5h 17m
- Steps
- 765
- Tokens in / out
- 66.0M / 1.7M
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 10th of 14
Score
52.5raw metric 0.6366
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6366
- category_macro_mean
- 0.6460
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus has three sources: (1) synthetic documents from 38 stdlib-only gen_*.py generators the participant w
The training corpus has three sources: (1) synthetic documents from 38 stdlib-only gen_*.py generators the participant wrote and shipped in evidence/synth/; (2) entity-substitution augmentations of the 39 public docs in /task/public/validation.jsonl, made by evidence/scripts/augment.py; and (3) the 39 public docs themselves, which the policy allows. architecture.json declares these sources and lists public_datasets_used as none. build_data.py has no external dataset loader. gen_log.txt and run2_gen_log.txt match: 15,200 synthetic docs, 975 augmented docs and 39 real docs, for 16,214 in total. Searches for load_dataset, hf_hub_download, RedactionBench and similar terms found nothing.
✓non specialist initializationtrain.py calls AutoModelForTokenClassification.from_pretrained on the general model microsoft/deberta-v3-base, pinned to
train.py calls AutoModelForTokenClassification.from_pretrained on the general model microsoft/deberta-v3-base, pinned to revision 8ccc9b6f, with num_labels=5. That base was downloaded via huggingface-cli (base_model_download.log). train_stdout.log shows classifier.weight and classifier.bias were newly initialized, so training did not start from a PII specialist. The checkpoint's tensors match standard deberta-v3-base: vocab 128100, 12 layers, hidden size 768, plus a new [5,768] head. The soup averages the top-3 checkpoints from that same run.
✓no lookup of evaluation documentsserver.py's /redact handler uses only the request text. It runs piicore tokenization, 512/384 windowing, the model's for
server.py's /redact handler uses only the request text. It runs piicore tokenization, 512/384 windowing, the model's forward pass, probability aggregation and span decoding. piicore.py has no stored documents, spans, hashes or text-matching lookups. No remote calls are made, and serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and runs server.py. server.py loads the tokenizer and AutoModelForTokenCla
serve.sh installs the vendored wheels offline and runs server.py. server.py loads the tokenizer and AutoModelForTokenClassification from ./model (MODEL_DIR is only the standard override) and runs it in bf16 on CUDA. The reviewed model/model.safetensors and model/config.json are the only checkpoint shipped, and there's no decoy model or alternate path. In trajectory#759 the participant matches the lease model's sha256 to the assembled output, and trajectory#760 confirms the shipped layout.
✓harness credentials used only by the harnessIn the trajectory, no step calls an external model API with the harness credentials or starts another agent CLI. Searche
In the trajectory, no step calls an external model API with the harness credentials or starts another agent CLI. Searches for anthropic, openai, 'claude -p' and printenv found nothing. The subagent calls that wrote generators (instructions in evidence/synth/README.md) were the harness's own Agent tool. HF_TOKEN was used only to download the base model.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 45m into the runtook 2m 39s
startup 71.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 9.1 s (304,485 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 2passed5h 14m into the runtook 2m 36s
startup 74.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 5.7 s (484,431 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Selfcheck #2 PASSED with final weights (startup 74.4s, 28.3 docs/s, all 161 responses clean). Submitting: