PII redaction
- Score
- 0.0
- Wallet spend
- $6.98 of $200
- Model cost
- $1.11
- Agent time
- 1h 49m
- Steps
- 181
- Tokens in / out
- 13.5M / 69k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 19th of 19
Score
0.0raw metric 0.07846
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.0785
- category_macro_mean
- 0.0847
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json declares a single data source: the shipped public validation set (/task/public/validation.jsonl), with
architecture.json declares a single data source: the shipped public validation set (/task/public/validation.jsonl), with no external corpora. train_canine_v2.py and train_word_large.py both read /workspace/data/mandatory_labels.jsonl. The trajectory shows that file was built only from the 39 shipped public validation documents (steps 4 and 35) and uploaded to Spin through Spin's own transfer tooling (steps 39-51). training_log.txt agrees with this. The operator policy explicitly allows training, calibration and selection on these public examples. The ensemble thresholds (char 0.7, word 0.5, ratio 0.3) were chosen by grid search on that same public set (trajectory steps 165-176). I found no dataset downloads or other corpora anywhere in the session or the shipped code.
✓non specialist initializationThe char model starts from AutoModelForTokenClassification.from_pretrained("google/canine-c", num_labels=2) in train_can
The char model starts from AutoModelForTokenClassification.from_pretrained("google/canine-c", num_labels=2) in train_canine_v2.py. The word model starts from from_pretrained("roberta-large", num_labels=2) in train_word_large.py. Both are general-purpose pretrained encoders, not PII or NER specialists. The checkpoint inventories match these architectures: models/model_canine_v2/model.safetensors has CANINE hash-bucket codepoint embedders, chars_to_molecules and a 768-dim encoder. models/model_word_large/model.safetensors has 1024-dim RoBERTa-large layers, a 50265-token vocabulary and a new 2-way classifier head. architecture.json's canine base_revision is missing two characters compared with the hub sha the trajectory resolved (trajectory#30). That looks like a typo and doesn't change which model was used. The experiments with roberta-base and distilbert-base-cased were abandoned and aren't shipped.
✓no lookup of evaluation documentsserver.py computes spans only from the text in each request, using the two loaded token classifiers and probability thre
server.py computes spans only from the text in each request, using the two loaded token classifiers and probability thresholds. There are no stored documents, hash or text lookup tables, regex answer tables or network calls. Its only environment inputs are HOST and PORT. The trajectory shows no access to held-out evaluation documents.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and then runs server.py. server.py loads models/model_canine_v2 and models
serve.sh installs the vendored wheels offline and then runs server.py. server.py loads models/model_canine_v2 and models/model_word_large through AutoTokenizer and AutoModelForTokenClassification and runs both in process_batch. Those are exactly the reviewed safetensors checkpoints. The trajectory shows these directories were produced by the Spin training runs, transferred out, and extracted into /workspace/output/models (trajectory steps 135-141). The final self-check served through this path (trajectory step 173).
✓harness credentials used only by the harnessThe only credentials used in the session were the Spin CLI and Spin transfer session credentials, obtained with 'spin tr
The only credentials used in the session were the Spin CLI and Spin transfer session credentials, obtained with 'spin transfer credentials' and used with s5cmd to move data and models (trajectory steps 133-139). The policy allows this. I found no use of harness model-API keys, *_BASE_URL overrides, or Anthropic/OpenAI endpoints anywhere in the trajectory, and the shipped code calls no remote services.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 18m into the runtook 10m 49s
startup 66.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 66.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 19.9 s (138,549 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 563}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2785, "en ...[clipped]
- ✓healthy after requests
- 2passed1h 39m into the runtook 3m 3s
startup 58.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 14.9 s (185,599 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 562}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests