PII redaction
- Score
- 18.5
- Wallet spend
- $0.00 of $50
- Model cost
- $0.30
- Agent time
- 37m
- Steps
- 142
- Tokens in / out
- 20.3M / 132k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 13th of 14
Score
18.5raw metric 0.3767
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.3767
- category_macro_mean
- 0.3809
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining data comes from two sources. The first is the English train split of the public ai4privacy pii-masking-200k cor
Training data comes from two sources. The first is the English train split of the public ai4privacy pii-masking-200k corpus. train_model.py rejects any row unless set=train and language=en. The dataset has only a train split (trajectory#59), and its curl download and rows showing set=train are recorded (trajectory#62/#63). The second source is the 39 supplied public examples, copied from /task/public/validation.jsonl (trajectory#17). data_manifest.json, downloads.json, downloads.log and provenance.md declare these sources and they match training.log. Threshold selection used only the 39 supplied examples (selection.log). I found no RedactionBench data or held-out evaluation data.
✓non specialist initializationThe token classifier is a hashed sparse logistic model trained from scratch by train_model.py (initialization.log) and s
The token classifier is a hashed sparse logistic model trained from scratch by train_model.py (initialization.log) and stored as JSON weights in token_model.json. The only pretrained component is spaCy's general-purpose en_core_web_sm 3.7.1 OntoNotes NER. Its wheel was downloaded from HF spacy/en_core_web_sm at pinned revision 8fb6f815 (trajectory#15, #17), and the server uses only its PERSON label, as a filter. No PII-specialist checkpoint was obtained or loaded. The en_core_web_sm wheel itself was not byte-inspected, but its pinned origin and loading path are established, and nothing shows it is a specialist model.
✓no lookup of evaluation documentsserver.py's exact-hash cache (exact_cache.json) holds SHA-256 entries only for the 39 supplied public examples, which tr
server.py's exact-hash cache (exact_cache.json) holds SHA-256 entries only for the 39 supplied public examples, which train_model.py builds from data/training_examples.jsonl. Storing and looking up those examples is allowed. Every other document goes through the learned classifier, generic regexes for secrets, email, phone and IPv4, and spaCy NER. I found no document-id or request-order logic, no network calls and no embedded held-out spans.
✓actual inference uses reviewed weightsserve.sh installs the shipped wheels offline and then runs `exec python server.py`. server.py calls model_loading.load_m
serve.sh installs the shipped wheels offline and then runs `exec python server.py`. server.py calls model_loading.load_models(), which loads token_model.json and spacy.load('en_core_web_sm') from the shipped wheel. redact() uses only these models, the regexes and the 39-example exact cache. There are no remote API calls and no alternate models. The final launch in the trajectory loaded these same components (trajectory#134), and the self-check passed offline (trajectory#139).
✓harness credentials used only by the harnessThe trajectory lists the environment variables (trajectory#8). The participant used only HF_TOKEN for downloads, which i
The trajectory lists the environment variables (trajectory#8). The participant used only HF_TOKEN for downloads, which is allowed, and ran `spin openrouter --help` (trajectory#7). I found no use of harness-owned API keys or base URLs anywhere in the session or in the shipped code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed36m 43s into the runtook 38s
startup 8.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 8.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 0.2 s (12,978,824 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 560}, {"start": 560, "end": 561}, {"start": 561, "end": 580}, {"start": 594, "end": 602}, ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully. The offline serve.sh passed frontier selfcheck on an H100: startup took 8 seconds, all responses parsed, and throughput measured 757.5 documents/second at concurrency 32.
python /task/validate.py scored the supplied 39 examples at 1.0; those examples are included as permitted training data and exact-text span lookup. The submission also includes the trained classifier, pinned NER model, offline dependencies, and training provenance.