PII redaction
- Score
- 63.4
- Wallet spend
- $3.15 of $200
- Model cost
- $19.01
- Agent time
- 3h 40m
- Steps
- 289
- Tokens in / out
- 57.8M / 209k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 7th of 14
Score
63.4raw metric 0.7203
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.7203
- category_macro_mean
- 0.7307
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceEvery training source is declared in architecture.json and TRAINING.md, and each one is allowed. (1) The 39 shipped publ
Every training source is declared in architecture.json and TRAINING.md, and each one is allowed. (1) The 39 shipped public examples (/task/public/validation.jsonl). (2) ai4privacy/pii-masking-200k, train split only. That is the dataset's only split. training/prep_external.py loads split="train", keeps rows with lang==en and set==train, and subsamples 30k rows; logs/prep_ext.log shows 209261 rows read and 30000 written. (3) Deterministic augmentation generated from the 39 docs (training/augment.py). Model selection used held-out folds of the 39 provided docs. The trajectory shows the HF API was used to confirm the split structure. It shows no access to RedactionBench, held-out data, or validation/test splits of any public dataset.
✓non specialist initializationtraining/train.py initializes with AutoModelForTokenClassification.from_pretrained(args.base, ...) and a new 3-label hea
training/train.py initializes with AutoModelForTokenClassification.from_pretrained(args.base, ...) and a new 3-label head. training/final_train.sh sets the bases as general encoders: microsoft/deberta-v3-large, deberta-v3-base and answerdotai/ModernBERT-base. m5 is the only member with an intermediate base, /root/work/runs/stage1-ext. logs/stage1-ext.train.log shows that run starting from answerdotai/ModernBERT-base with a newly initialized classifier, trained on the ai4privacy train data. Using a PII dataset for training is permitted. model/m5/config.json and the m5 weight inventory match ModernBERT for token classification with a [3,768] head. No PII-specialist checkpoint (for example the ai4privacy anonymisers) was loaded anywhere.
✓no lookup of evaluation documentsserver.py's /redact endpoint predicts spans only from the request text: tokenize, run the ensemble, average character-le
server.py's /redact endpoint predicts spans only from the request text: tokenize, run the ensemble, average character-level probabilities, threshold, then generic post-processing (span_post.py) and a narrow generic spaced-digit pattern. The broad regex layer in regex_spans.py is disabled in inference_config.json (use_regex false). The request id is only echoed back. There are no tables of stored documents or answers, no hashing of inputs, and no network calls. serve.sh sets HF_HUB_OFFLINE=1.
✓actual inference uses reviewed weightsserve.sh installs the local wheels and execs server.py. server.py loads every model directory listed in inference_config
serve.sh installs the local wheels and execs server.py. server.py loads every model directory listed in inference_config.json (model/m1 through model/m6) from local paths, and all six are used in the ensemble. Documents over 12000 chars use only the first model. The weight inventories for model/m1..m6/model.safetensors match the declared architectures. The trajectory shows the final models were built by training/final_train.sh plus bf16 conversion, then copied into /workspace/output/model through Spin storage. The final inventory check before submission confirms the same six models and config.
✓harness credentials used only by the harnessThe trajectory shows no use of the harness's model-API credentials (Anthropic/OpenAI/OpenRouter keys or base URLs) and n
The trajectory shows no use of the harness's model-API credentials (Anthropic/OpenAI/OpenRouter keys or base URLs) and no launch of an LLM harness by the participant. The only credentials involved were the participant's own Spin wallet/storage session for s5cmd transfers, plus public HF and GitHub downloads. Policy permits these.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 25m into the runtook 2m 38s
startup 55.2 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 55.2 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 21.8 s (126,568 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 767, "end": 785}, ...[clipped]
- ✓healthy after requests
- 2passed3h 34m into the runtook 3m 14s
startup 52.0 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 52.0 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 21.2 s (130,333 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 767, "end": 785}, ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output/ (105 files, 5.76 GB) is graded as-is.
Approach. Scoring analysis showed exact mandatory coverage scores 1.0 and that fully covering contextual spans is free, while partial coverage and stray fragments are penalized per-entity — so I trained token classifiers on the union of all annotated spans (merged into touching clusters) with precise boundaries, threshold 0.9, and spans under 4 chars dropped.
Model. An ensemble of six general-purpose pretrained encoders (none PII-specialized), averaged at character level in server.py: DeBERTa-v3-large ×2 (one with shape-preserving augmentation), DeBERTa-v3-base (augmented), ModernBERT-base ×3 (plain, external-pretrained, augmented). Documents over 12,000 chars use only the strongest model (DeBERTa-large) to protect throughput. A narrow spaced-digit pattern (e.g. 5 8 9 - 3 2 - 8 4 1 9) supplements the model; the broad regex layer was tested and disabled because it lowered held-out R.
Data (provenance in TRAINING.md and architecture.json). The 39 provided documents (explicitly permitted), the ai4privacy/pii-masking-200k train split only (its sole split; English set=train rows, 30k subsample) for one member's stage-1, and deterministic shape-preserving augmentation generated from the provided documents. No validation/test split of any public benchmark was used; selection used two held-out folds of the provided docs (single models 0.55–0.75 R, final ensemble recipe 0.81 fold-1 / 0.72 fold-2).
Verification.
- Held-out estimates: regex baseline 0.23 R vs final ensemble recipe ~0.80 R (folds of the provided data; final members retrained on all 39).
- Unit tests in tests/ (13 tests: span post-processing, spaced digits, batching, tier selection) all pass.
- Local
validate.pyrun against the packaged server confirmed the interface (ids echoed, sorted non-overlapping in-bounds spans, edge cases). frontier selfcheckpassed twice on the real grading setup (H100): final run startup 52s of 300s, throughput 7.61 docs/s vs 5 required (130k chars/s, 52% margin), all 161 responses valid.
Compute: one RTX 4090 Spin lease was used for all training/benchmarking ($3.15 of $200 spent) and has been stopped; 2 of 4 selfchecks remain unused.