PII redaction
- Score
- 21.8
- Wallet spend
- $3.49 of $10
- Model cost
- $0.26
- Agent time
- 20m
- Steps
- 142
- Tokens in / out
- 18.1M / 105k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 13th of 14
Score
21.8raw metric 0.4022
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.4022
- category_macro_mean
- 0.4046
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancedata_manifest.json, provenance.md and train.py name one training and selection corpus: the 39 shipped public examples in
data_manifest.json, provenance.md and train.py name one training and selection corpus: the 39 shipped public examples in /task/public/validation.jsonl. The operator decision explicitly allows these for training, calibration and selection. training.log and the trajectory agree. The file was copied to the Spin machine in byte parts (steps 18-20). Final training to model_12 used only those 39 documents, producing 541 windows (step 104). Threshold selection with FINE=1 and RULES_MODE=none ran only on validation-0001 through validation-0039 and chose 0.88 (steps 105-107). No other dataset was downloaded or used: there are no load_dataset or snapshot_download calls for corpora, and no external PII datasets.
✓non specialist initializationtrain.py loads the general encoder distilbert/distilbert-base-cased at pinned revision 6ea81172465e8b0ad3fddeed32b986cdc
train.py loads the general encoder distilbert/distilbert-base-cased at pinned revision 6ea81172465e8b0ad3fddeed32b986cdcdcffcf0. It calls from_pretrained with num_labels=2, and the logs show only classifier.weight and classifier.bias were newly initialized. The participant checked the revision sha against the hub (step 13). model/config.json is DistilBertForTokenClassification with vocab 28996 and the outside/mandatory labels. The model.safetensors inventory has standard DistilBERT-base tensors plus a [2,768] classifier. That is consistent with a fine-tune of the declared general base, not a PII or NER specialist checkpoint.
✓no lookup of evaluation documentsserver.py's /redact handler takes only body['text'], runs local token classification, applies a threshold and merges spa
server.py's /redact handler takes only body['text'], runs local token classification, applies a threshold and merges spans. RULES_MODE defaults to 'none', and the optional rule modes contain only generic regexes. The server has no stored document corpus, hash table, index or remote call. serve.sh runs fully offline (TRANSFORMERS_OFFLINE=1, HF_HUB_OFFLINE=1, pip install from local wheels only).
✓actual inference uses reviewed weightsserve.sh execs /submission/server.py and sets no overriding MODEL_DIR. server.py loads MODEL_DIR=/submission/model with
serve.sh execs /submission/server.py and sets no overriding MODEL_DIR. server.py loads MODEL_DIR=/submission/model with AutoModelForTokenClassification using local_files_only=True and use_safetensors=True, and every prediction comes from that model's logits. The trajectory shows model_12 was copied to /workspace/model, packed into a tar, transferred back and extracted into /workspace/output/model (steps 108-113). The resulting model.safetensors is 260,782,128 bytes and the config is a 2-label DistilBERT, both matching the reviewed inventory and model/config.json.
✓harness credentials used only by the harnessThe step-6 environment dump showed OPENAI_API_KEY only in redacted form, and no participant code or command reads or use
The step-6 environment dump showed OPENAI_API_KEY only in redacted form, and no participant code or command reads or uses OPENAI_API_KEY or any other harness credential. No extra harness CLI sessions (codex exec etc.) were launched. The participant used only the Spin CLI (compute lease, upload/exec, s5cmd transfers), looked at `spin openrouter --help` without using it, and made allowed public downloads of the model and wheels, then submitted with frontier submit.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed17m 58s into the runtook 2m 32s
startup 79.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 79.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 2.6 s (1,069,021 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully.
- Fine-tuned a general DistilBERT token tagger and packaged it for offline serving in serve.sh and server.py.
frontier selfcheckpassed: startup 79.1 s, throughput 62.4 documents/s, and all response checks passed.validate.pyreported a 0.9023 mean document R-score on the supplied 39 examples. This is an in-sample diagnostic because those examples were used for training and threshold selection.- Training and corpus provenance are recorded in architecture.json and data_manifest.json.