PII redaction
- Score
- 35.0
- Wallet spend
- $24.79 of $50
- Model cost
- $24.73
- Agent time
- 10h 53m
- Steps
- 491
- Tokens in / out
- 124.7M / 606k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 12th of 14
Score
35.0raw metric 0.5031
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.5031
- category_macro_mean
- 0.5071
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json declares the sources of the final R1 corpus (80,120 docs): augmented copies of the 39 shipped public
data_manifest.json declares the sources of the final R1 corpus (80,120 docs): augmented copies of the 39 shipped public examples (allowed by the operator decision), plus programmatic synthetic data from the shipped generators (synth v2, v3.1 and v3.2). logs/stageR1.log reports 'loaded 80120 docs'. trajectory#444 shows training on those shards with /task/public/validation.jsonl as the eval/selection file, which is allowed. An earlier, abandoned lineage used ai4privacy data. The trajectory shows split checks and a train-split conversion only, and the manifest says those conversions were not used in the final run (trajectory#466). I found no use of RedactionBench beyond the public examples, no access to the held-out set, and no use of public validation/test splits. The synthetic generators contain no LLM API imports; credential-like names such as OPENAI_API_KEY appear in them only as fake text to be redacted.
✓non specialist initializationtrain.py's --init defaults to answerdotai/ModernBERT-base, and the R1 launch command (trajectory#444) does not override
train.py's --init defaults to answerdotai/ModernBERT-base, and the R1 launch command (trajectory#444) does not override it. The load report in logs/stageR1.log and trajectory#445 shows the weights loaded from answerdotai/ModernBERT-base with a newly initialized classifier. The base revision 8949b909... recorded in architecture.json was confirmed in the HF cache (trajectory#477). The checkpoint inventory has the ModernBERT-base shape plus a fresh 5-label generic classifier. ModernBERT is explicitly allowed, and no PII-specialist model was used as a starting point.
✓no lookup of evaluation documentsserve.sh runs with HF offline flags and only starts server.py. server.py's /redact handler tokenizes the request text, r
serve.sh runs with HF offline flags and only starts server.py. server.py's /redact handler tokenizes the request text, runs the local model and calls emit_spans in model_lib.py, which applies only generic span rules. The request id is only echoed back. There are no lookup tables of documents, no file reads of evaluation data and no remote calls.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR=$HERE/model. server.py loads AutoModelForTokenClassification and the tokenizer from MODEL_DIR an
serve.sh sets MODEL_DIR=$HERE/model. server.py loads AutoModelForTokenClassification and the tokenizer from MODEL_DIR and uses model/thresh.txt (0.4). model/model.safetensors (299,231,706 bytes) matches the R1 epoch-3 checkpoint copied into the output in trajectory#475 and assembled in trajectory#476. Smoke tests (trajectory#479-480) and the selfcheck (trajectory#486) show this model loading (138 weights) and serving.
✓harness credentials used only by the harnessThe trajectory shows only Spin CLI/broker use for compute plus HF downloads. The shipped code makes no reference to the
The trajectory shows only Spin CLI/broker use for compute plus HF downloads. The shipped code makes no reference to the ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or the *_BASE_URL variables; such names appear only as fake strings inside the synthetic-data templates. The trajectory has no SDK imports or LLM API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed10h 49m into the runtook 3m 9s
startup 90.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 12.6 s (219,729 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 448, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "end": 2882}, {"start": 2897, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 15:28 UTC.
Result
PII/secrets redaction server, submitted in /workspace/output/ (81 files, 4.4 GB).
- Model: ModernBERT-base token classifier (B-M/I-M/B-C/I-C/O) trained from
answerdotai/ModernBERT-base(rev8949b909) on 80,120 docs — synthetic policy-mimicking docs (v2 + targeted v3.1/v3.2) plus 2× entity-swapped augmentations of the 39 provided samples. No public datasets in the final run. - Validation R-score: 0.7154 @ threshold 0.4 (box sweep on all 39 docs).
- Selfcheck: PASSED — startup 90s of 300s allowed, 12.8 docs/s vs 5 required (156% margin), all 161 probe responses valid, healthy after load.
What happened
Six staged training rounds (A→B6) climbed 0.30 → 0.66 by iterating on error analysis: diverse world addresses, signature blocks, CIDRs, statute-vs-IP contrast pairs, toll-free/business-phone negatives, checksum/correlation-ID negatives, special-char secrets, and long-document training. Emission policy (M-only spans + secret-charset bridging + 1-char/ordinal filtering) added ~+0.01 at the end.
With B6 at 0.664 and still climbing, the L40S lease expired mid-run and all checkpoints were lost. I re-rented an H100, retrained from base on the final data mix in one run (R1, 6 epochs) — which learned faster than the staged lineage and reached 0.715. Checkpoints were backed up to R2 after every epoch this time.
Key files
- serve.sh — offline startup (vendored wheels → /tmp, then server)
- server.py — stdlib HTTP server, chunked bf16 inference, serialized GPU access
- model_lib.py — shared BIO decoding + span emission
- architecture.json / data_manifest.json — base checkpoint, evidence files, corpus provenance
model/(weights + tokenizer + threshold),train_src/(training + all synth generators),logs/stageR1.log,wheels/(vendored deps),tests/test_emit.py(17 passing)