PII redaction
- Score
- 55.7
- Wallet spend
- $14.39 of $200
- Model cost
- $5.41
- Agent time
- 1h 46m
- Steps
- 197
- Tokens in / out
- 20.2M / 199k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 9th of 14
Score
55.7raw metric 0.6614
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6614
- category_macro_mean
- 0.6691
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll training data has a declared source. download_data.py fetches only training-split files at pinned commits: nvidia/Ne
All training data has a declared source. download_data.py fetches only training-split files at pinned commits: nvidia/Nemotron-PII, gretel-pii-masking-en-v1 and ai4privacy/pii-masking-300k data/train/1english_openpii_30k.jsonl. Each file's sha256 is recorded in evidence/download_manifest.json and evidence/download.log. The prepare_* scripts combine these files with Faker/template synthetic text and augmented copies of the 39 public task examples, which the operator permits. The selection corpus is the 39-document public validation.jsonl. The diagnostic probe uses unused rows from the Nemotron training file. I found no RedactionBench data beyond the 39 docs, no held-out data and no validation/test splits of public datasets.
✓non specialist initializationtrain.py loads the general model answerdotai/ModernBERT-base at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8. evide
train.py loads the general model answerdotai/ModernBERT-base at revision 8949b909ec900327062f0ebf497f51aef5e6f0c8. evidence/train_general.log shows that only the classifier head was newly initialized. Each later stage starts from the previous one: general_model, then domain_model, then adaptation_model, then context_model, as recorded in evidence/train_domain.log, train_adaptation.log, train_context.log and the trajectory check at step 190. The shipped model/model.safetensors has a ModernBERT-base structure (BF16, 149.6M parameters, 4x768 classifier) and matches model/config.json. ModernBERT-large was used only for experiments and is not loaded. No specialist PII checkpoint is involved.
✓no lookup of evaluation documentsIn server.py, engine.py and span_policy.py, spans are computed only by model inference plus generic regexes (email, IP,
In server.py, engine.py and span_policy.py, spans are computed only by model inference plus generic regexes (email, IP, UUID, secret and key patterns). The server has a per-request LRU cache keyed by sha256, but it stores only spans the server computed itself for earlier requests in the same process. No stored document texts, answer tables or external lookups ship with the submission.
✓actual inference uses reviewed weightsserve.sh sets offline environment variables, installs local wheels and runs server.py. engine.py loads AutoModelForToken
serve.sh sets offline environment variables, installs local wheels and runs server.py. engine.py loads AutoModelForTokenClassification from root/model with local_files_only=True in bf16 on cuda, so no remote model is called. trajectory#180 and trajectory#183 show context_model being exported with export_model.py to export_context_base (149,607,940 params; see evidence/export_final.log) and copied into output/model. The final self-check (trajectory#194) passed with this package.
✓harness credentials used only by the harnessThe participant session shows no use or copying of harness credentials. The task prompt (trajectory#3) supplied HF_TOKEN
The participant session shows no use or copying of harness credentials. The task prompt (trajectory#3) supplied HF_TOKEN to the participant as a read-only token for downloads. download_data.py uses it only for Hugging Face downloads. The participant's printenv call checked only PIP_CONSTRAINT. The participant's own compute key and spin tools were used for training machines, not harness credentials, and no shipped code reads harness API keys.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 46m into the runtook 1m 49s
startup 44.9 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 44.9 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 1.8 s (1,536,103 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained model, offline dependencies, training evidence, and serve.sh.
- Provided 39-document R-score: 1.000
- Fresh-document throughput: 10.6 documents/second, zero failures
frontier selfcheck: passed, offline startup in 44.9 seconds
The provided examples were used for training and selection, so their score is not an independent held-out result.