PII redaction
- Score
- 50.1
- Wallet spend
- $10.47 of $50
- Model cost
- $2.16
- Agent time
- 3h 50m
- Steps
- 303
- Tokens in / out
- 26.2M / 83k
- Self-checks
- 1 passed of 4 used
- Rank at $50 wallet
- 13th of 19
Score
50.1raw metric 0.6182
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6182
- category_macro_mean
- 0.6293
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe training corpus is declared in data_manifest.json and in training/train.py. It is the public ai4privacy/pii-masking-
The training corpus is declared in data_manifest.json and in training/train.py. It is the public ai4privacy/pii-masking-200k dataset, loaded with load_dataset(..., split='train'), shuffled with seed 42, cut to the first 40k and filtered by label. The only other data is the 39 shipped public validation examples, oversampled 10x. The operator explicitly allows training on those. The trajectory shows the actual train split being downloaded (step 50), validation.jsonl being uploaded (step 46) and the final training run (steps 110-111). Its losses match training/train.log, which calls itself a reconstructed summary. No evaluation or hidden data was used.
✓non specialist initializationtrain.py starts from AutoModelForTokenClassification.from_pretrained('microsoft/deberta-v3-base', num_labels=2), a gener
train.py starts from AutoModelForTokenClassification.from_pretrained('microsoft/deberta-v3-base', num_labels=2), a general-purpose pretrained encoder. The 2-label classifier head is newly initialized, and the run log reports the classifier weights as missing. model/config.json (DebertaV2ForTokenClassification, 768 hidden, 12 layers, 128100 vocab) and the safetensors inventory (deberta.* tensors plus classifier [2,768]) are consistent with that. The trajectory shows no specialist PII model being downloaded or used as the starting point.
✓no lookup of evaluation documentsserve.py tokenizes only the incoming request text, runs the token classifier and turns the mask into spans with simple w
serve.py tokenizes only the incoming request text, runs the token classifier and turns the mask into spans with simple whitespace merging. It has no stored documents, no answer lookup tables, no regex memorization of evaluation content and no network calls. serve.sh only does an offline install of wheels and then starts serve.py.
✓actual inference uses reviewed weightsserve.sh runs serve.py. serve.py loads AutoModelForTokenClassification and its tokenizer from BASE_DIR/model (model/mode
serve.sh runs serve.py. serve.py loads AutoModelForTokenClassification and its tokenizer from BASE_DIR/model (model/model.safetensors, config.json, tokenizer files), and the prediction comes from that model's logits alone. The trajectory shows the trained /tmp/pii-model2 checkpoint being copied into output/model (steps 116, 192). No remote model or decoy path appears.
✓harness credentials used only by the harnessThe trajectory shows no use of harness API keys and no extra harness CLIs being launched. HF_TOKEN was used for Hugging
The trajectory shows no use of harness API keys and no extra harness CLIs being launched. HF_TOKEN was used for Hugging Face downloads, including on the Spin machine, which is allowed (steps 52-53). Model transfer used the Spin session's own S3/s5cmd credentials (step 115).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 47m into the runtook 3m 14s
startup 77.8 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents u
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.8 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 2failed2h 55m into the runtook 3m 18s
startup 70.4 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents u
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.4 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 3failed3h 36m into the runtook 4m 53s
startup 68.2 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents u
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.2 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 4passed3h 47m into the runtook 2m 15s
startup 65.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 25.7 s (107,386 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2908}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submission confirmed. The solution is complete.