PII redaction
- Score
- 0.0
- Wallet spend
- $12.22 of $50
- Model cost
- $17.80
- Agent time
- 1h 20m
- Steps
- 112
- Tokens in / out
- 24.1M / 109k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 14th of 14
Batch exceeded 32.2s deadline; score is zero
Score
0.0
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training corpus has a declared source. (1) The ai4privacy/pii-masking-200k english_pii_43k.jsonl train split is li
Every training corpus has a declared source. (1) The ai4privacy/pii-masking-200k english_pii_43k.jsonl train split is listed in data_manifest.json with a curl command. download_log.txt records that command and the file size. One small issue: the logged URL uses resolve/main instead of the pinned revision, so the exact revision is recorded as unverified, but this is not a violation. (2) The 39 public validation docs, which the task explicitly allows. (3) Synthetic templates from synth.py, which only reuses values from ai4privacy train rows plus randomly generated secrets. train.py, train2.py, train3.py and train4.py only read these sources. No held-out or evaluation-set data appears.
✓non specialist initializationtrain.py starts from the general MLM base answerdotai/ModernBERT-base (pinned revision in architecture.json) with num_la
train.py starts from the general MLM base answerdotai/ModernBERT-base (pinned revision in architecture.json) with num_labels=3. The load report in training_log.txt shows classifier.weight and classifier.bias as MISSING (newly initialized) and decoder.bias as UNEXPECTED, which is what a generic MLM checkpoint looks like, not a PII-specialist one. Later stages only continue from these earlier checkpoints. model/config.json and the model.safetensors tensor inventory match a ModernBERT-base token classifier with a [3,768] classifier head.
✓no lookup of evaluation documentsserver.py passes the request text to REDACTOR.redact. redact.py produces spans from sliding-window model probabilities w
server.py passes the request text to REDACTOR.redact. redact.py produces spans from sliding-window model probabilities with a threshold, then merges them with generic regex rules (credentials, assignments, ARNs, phones, DOB, IPs with context, etc.). There are no stored spans, document fingerprints, hash or id-keyed behavior, retrieval or remote calls.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, installs wheels with --no-index and runs server.py. redact.py
serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, installs wheels with --no-index and runs server.py. redact.py loads the tokenizer and AutoModelForTokenClassification from MODEL_DIR, which defaults to ./model. Only one checkpoint is shipped (model/model.safetensors with a matching config.json), and the code contains no alternate model path or network inference. In the trajectory, model4 scored lower than model2 and was not selected, which matches training_log.txt.
✓harness credentials used only by the harnessThe trajectory shows GPU work done through the provided spin compute exec/upload lease, and HF_TOKEN used only for read-
The trajectory shows GPU work done through the provided spin compute exec/upload lease, and HF_TOKEN used only for read-only downloads. No step reads or uses the harness's model API keys or base URL, calls a model API directly, or launches another agent harness. The API_KEY/BASE_URL strings that appear are only regex patterns or validation-doc text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 3m into the runtook 3m 15s
startup 88.5 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 88.5 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 15.4 s (179,133 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 519, "end": 520}, {"start": 522, "end": 525}, {"start": 526, "end": 527}, {"start": 530, "end": 531}, {"start": 548, "end": 583}, {"start": 594, "end": 612}, {"start": 614, "end": 624 ...[clipped]
- ✓healthy after requests
- 2passed1h 13m into the runtook 3m 16s
startup 99.8 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 99.8 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.7 s (164,990 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 519, "end": 520}, {"start": 522, "end": 525}, {"start": 526, "end": 527}, {"start": 530, "end": 531}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 634 ...[clipped]
- ✓healthy after requests
- 3passed1h 16m into the runtook 2m 43s
startup 74.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 16.2 s (170,689 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 519, "end": 520}, {"start": 522, "end": 525}, {"start": 526, "end": 527}, {"start": 530, "end": 531}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 634 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The redaction server is submitted. It finds personal identifiers, contact and account details, credentials, and other secrets, and returns the character spans to redact.
The server is a ModernBERT token classifier, started from answerdotai/ModernBERT-base, fine-tuned on the English training split of ai4privacy/pii-masking-200k plus the 39 documents you provided. High-precision patterns cover secrets, structured identifiers, signatures, and seed-phrase words. POST /redact returns sorted, non-overlapping character spans.
On your 39 documents the mean document R-score is 0.729. The graded selfcheck came up healthy in 75 seconds and handled about 10 documents per second, above the 5-per-second requirement.