PII redaction
- Score
- 17.4
- Wallet spend
- $0.00 of $200
- Model cost
- $0.18
- Agent time
- 17m
- Steps
- 99
- Tokens in / out
- 10.9M / 97k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 14th of 14
Score
17.4raw metric 0.3686
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.3686
- category_macro_mean
- 0.3676
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe only corpus used for training, calibration and threshold selection is training_data/validation.jsonl. It is a byte-i
The only corpus used for training, calibration and threshold selection is training_data/validation.jsonl. It is a byte-identical copy of the 39 shipped public examples from /task/public/validation.jsonl, and the trajectory confirms the matching SHA-256. The source is declared in data_manifest.json and provenance.log. train_model.py and calibrate_fallback.py read only this file. The operator policy explicitly allows the public examples for training, calibration and selection. Neither the code nor the trajectory pulls any external, held-out or benchmark-test data.
✓non specialist initializationtrain_model.py trains a sparse-feature token logistic regression from zero-initialized weights with a fixed seed (6 epoc
train_model.py trains a sparse-feature token logistic regression from zero-initialized weights with a fixed seed (6 epochs of SGD). architecture.json declares base_model none / from-scratch, and model/trained.json holds only the resulting feature weights, bias and threshold. In the trajectory the participant tried spaCy en_core_web_sm, a general-purpose NER model rather than a PII specialist, then abandoned it; it was never shipped and the final listing contains no wheels or other checkpoints. No PII-specialist initialization is used.
✓no lookup of evaluation documentsmodel/known.json is an exact SHA-256 index of only the 39 shipped public documents, built by train_model.py; training.lo
model/known.json is an exact SHA-256 index of only the 39 shipped public documents, built by train_model.py; training.log reports exact_documents=39. Storing and looking up those public examples is explicitly permitted. For any other text, redact_model.py combines generic regex and field rules with classifier spans computed on the request text alone. There is no network access, no retrieval of evaluation documents and no hidden-label source.
✓actual inference uses reviewed weightsserve.sh runs server.py from the output directory. server.py creates RedactionModel(), and the POST handler calls redact
serve.sh runs server.py from the output directory. server.py creates RedactionModel(), and the POST handler calls redact(MODEL, text). redact_model.py loads model/trained.json and model/known.json, the same artifacts that train_model.py produced and that were reviewed. There is no alternate model path, no remote inference call and no decoy weights, and the trajectory's server startup reported known=39 and feature_weights=38832, matching training.log.
✓harness credentials used only by the harnessThe trajectory shows no use of the harness API keys or base URLs by the participant's code or commands. No extra agent o
The trajectory shows no use of the harness API keys or base URLs by the participant's code or commands. No extra agent or harness CLI was launched, and nothing was uploaded to any external service. The only external fetch was the abandoned spaCy wheel download from public GitHub releases, which used no harness credentials. Submission and selfcheck went through the provided tooling.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed17m 40s into the runtook 20s
startup 0.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 0.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 0.3 s (10,368,013 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 560}, {"start": 560, "end": 561}, {"start": 561, "end": 580}, {"start": 594, "end": 602}, ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully.
The selfcheck passed on an H100: startup took 0.3 seconds, all 161 responses were valid, and throughput was 605 documents/second at concurrency 32. The supplied 39-document validation set scored 1.0.