PII redaction
- Score
- 60.5
- Wallet spend
- $5.76 of $10
- Model cost
- $4.27
- Agent time
- 1h 13m
- Steps
- 150
- Tokens in / out
- 17.1M / 148k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 7th of 14
Score
60.5raw metric 0.6981
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.6981
- category_macro_mean
- 0.7128
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe submission declares three training sources in architecture.json, and they match prepare_data.py and evidence/prepare
The submission declares three training sources in architecture.json, and they match prepare_data.py and evidence/prepare_data_v2.py. (1) ai4privacy/pii-masking-200k english_pii_43k.jsonl at the pinned revision 7c22e681. The code drops every row where row['set'] != 'train', so only the train split is used. (2) Locally generated Faker templates. (3) The 39 shipped public examples plus format-preserving augmentations of them; the operator allows these for training, calibration and selection. calibrate.py and select_model.py score only the provided rows, using 3-fold CV over the 39 documents. No public validation or test split or extra RedactionBench data is used anywhere. The session searches for RedactionBench, load_dataset and validation filters found nothing. The download commands are kept in download_sources.py, and the trajectory shows the pinned downloads (trajectory#16) and the split-count inspection (trajectory#19).
✓non specialist initializationBoth models start from general-purpose encoders: answerdotai/ModernBERT-large and ModernBERT-base, at pinned revisions d
Both models start from general-purpose encoders: answerdotai/ModernBERT-large and ModernBERT-base, at pinned revisions downloaded via snapshot_download (download_sources.py, trajectory#16). train.py calls from_pretrained(args.base, num_labels=5). evidence/train_large_stage1.log shows initialization from /workspace/base_large with a newly initialized classifier, then the chain runs stage1 → stage2 → stage3 checkpoint-452, which evidence/final_weights.json exports. The weight inventories and configs show plain ModernBertForTokenClassification models: 28 layers at hidden size 1024, and 22 layers at hidden size 768, each with a 5-label O/B-M/I-M/B-C/I-C head. ai4privacy is used only as a dataset, never as a model checkpoint.
✓no lookup of evaluation documentsThe serving path (server.py → predictor.py → decoding.py) tokenizes payload['text'], runs the token classifiers over sli
The serving path (server.py → predictor.py → decoding.py) tokenizes payload['text'], runs the token classifiers over sliding windows, and decodes with thresholds plus generic rules: extending email spans, a few secret-key regexes, and trimming IP:port brackets. The id field is only echoed back. There is no stored text, hash table, or logic keyed on document content or id, and no network access.
✓actual inference uses reviewed weightsserve.sh installs local wheels and runs exec python -S server.py. server.py builds a Predictor from ROOT/'model' and, be
serve.sh installs local wheels and runs exec python -S server.py. server.py builds a Predictor from ROOT/'model' and, because inference_config.json sets ensemble true, also from ROOT/'aux_model'. predictor.py loads both with AutoModelForTokenClassification.from_pretrained(local_files_only=True) and averages their probabilities with weight 0.5. The reviewed weights model/model.safetensors (791,691,130 bytes) and aux_model/model.safetensors (299,231,706 bytes) match the sizes recorded in evidence/final_weights.json. There are no remote inference calls and no alternative decoy path.
✓harness credentials used only by the harnessIn the trajectory, API_KEY and BASE_URL appear only in Spin documentation output (trajectory#6), and openrouter appears
In the trajectory, API_KEY and BASE_URL appear only in Spin documentation output (trajectory#6), and openrouter appears only in Spin help text. The participant used Spin to lease an H100 for compute (trajectory#11) and the HF read-only token to download models and data (trajectory#16), both of which are allowed. No step copies or uses harness API keys or base URLs in participant code, and no extra harness CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed19m 16s into the runtook 2m
startup 46.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 46.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 3.8 s (732,949 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
- 2passed1h 9m into the runtook 3m 47s
startup 57.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 57.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 8.3 s (331,048 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 2742, "end": 2757}, {"start": 2772, "end": 2804}, {"start": 2867, "end": 2882}, {"start": 2897, "end": 2932}, {"start": 2996, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the trained model, offline server, weights, dependencies, and provenance logs.
- Mean document R-score: 1.0 on the 39 provided examples, which were used for training.
- Final offline self-check: passed.
- Startup: 57 seconds.
- Throughput: 19.3 documents/second, with all 161 responses valid.