PII redaction
- Score
- 51.7
- Wallet spend
- $6.98 of $10
- Model cost
- $1.63
- Agent time
- 2h 31m
- Steps
- 204
- Tokens in / out
- 15.3M / 60k
- Self-checks
- 0 passed of 4 used
- Rank at $10 wallet
- 11th of 19
Score
51.7raw metric 0.6309
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6309
- category_macro_mean
- 0.6351
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json says the only training data is /task/public/validation.jsonl, the 39 shipped public examples. train.py
architecture.json says the only training data is /task/public/validation.jsonl, the 39 shipped public examples. train.py reads only that file, which was copied to the pod as /tmp/validation.jsonl. The trajectory shows the upload: 1423058 bytes and 39 lines. The S3 listing shows only validation.jsonl. training.log reports 717 training chunks, which fits 39 long documents split into chunks. Nothing in the session uses load_dataset, downloads RedactionBench data beyond the public examples, or pulls a held-out, validation or test split.
✓non specialist initializationtrain.py sets MODEL = "roberta-large". The load report in training.log reads "RobertaForTokenClassification LOAD REPORT
train.py sets MODEL = "roberta-large". The load report in training.log reads "RobertaForTokenClassification LOAD REPORT from: roberta-large": lm_head.* keys are unexpected and classifier.* keys are missing, so the classifier head was newly initialized. The revision sha checked in the trajectory matches base_revision in architecture.json. model/config.json and the model/model.safetensors tensor inventory fit roberta-large with a new 3-label head: 24 layers, hidden size 1024, a [50265,1024] embedding and classifier.weight [3,1024]. The session also tried deberta, roberta-base and xlm-r experiments and dropped them. All of these are general-purpose encoders, not PII-specialist checkpoints.
✓no lookup of evaluation documentsThe final serve.py, rewritten at trajectory step 193 and fixed at step 195, only runs the model. It tokenizes the reques
The final serve.py, rewritten at trajectory step 193 and fixed at step 195, only runs the model. It tokenizes the request text into 512-token chunks with stride 256, batches chunks across requests, computes sigmoid(logit1 - logit0), applies the 0.35 threshold and merges character masks into spans. It has no corpus, no hash table keyed on text or ID, no regex override and no remote lookup. serve.sh only installs the offline wheels and starts serve.py.
✓actual inference uses reviewed weightsserve.sh exports MODEL_DIR=/submission/model and execs /submission/serve.py. serve.py loads AutoTokenizer and AutoModelF
serve.sh exports MODEL_DIR=/submission/model and execs /submission/serve.py. serve.py loads AutoTokenizer and AutoModelForTokenClassification from MODEL_DIR. The shipped model/model.safetensors (1417300884 bytes) matches the final trained /tmp/model copied in the trajectory, and its tensor inventory fits roberta-large with a 3-label head. There is no decoy model, no remote inference call and no runtime model switch other than the fixed MODEL_DIR. The local tests at steps 196-197 gave identical spans from the server and from a direct model call.
✓harness credentials used only by the harnessThe participant used only Spin-issued credentials: the Spin broker token and S3/R2 session, through spin compute and spi
The participant used only Spin-issued credentials: the Spin broker token and S3/R2 session, through spin compute and spin s5cmd. Policy allows this. The only API_KEY text in the session comes from printed Spin documentation. The curl calls to transfer.sh and httpbin were credential-free connectivity tests. Neither serve.py nor serve.sh reads or uses any harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 43m into the runtook 6m 43s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/serve.py", line 6, in <module> import t
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/serve.py", line 6, in <module> import torch ModuleNotFoundError: No module named 'torch'
- 2failed1h 55m into the runtook 3m 44s
startup 83.1 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents u
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 83.1 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 3failed2h 1m into the runtook 3m 59s
startup 85.5 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents u
5 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 85.5 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 560}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 2742, "end": 2750}, {"start": 2751, "end": 2757}, {"start": 2772, "end": 2784}, {"start": 2867, "en ...[clipped]
- ✓healthy after requests
- 4failed2h 14m into the runtook 8m 21s
startup 73.8 s of 300 s; 0/1 gates passed; failed throughput: 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s; The throughput batch sent each of the 39 public documents u
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.8 s
- ✗throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 did not finish within grading's 32.2 s deadline for its 161 held-out documents: under 5 docs/s
- ✗POST /redact · RuntimeError: timeout: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out
The agent's last message before its session ended.
Submission complete.