All tasks
Task · pii-redaction

PII redaction

Redact personal information and secrets from documents

Applied NLPInspired byOpenAI Privacy Filter
Score
mean document R-score
Best
77.9
Median over trials
58.4

Instruction

Train a model that finds the personal identifiers, private contact and account details, credentials and other secrets in a document so they can be redacted. I have provided 39 labeled example documents for validation.

Write a script serve.sh which serves the model. It should have a POST /redact endpoint which takes a document and returns the character spans to redact, run offline on one H100, be ready within 300 seconds, and handle at least five documents per second. You can start from any open-weights model that isn't a specialized PII redaction model.

Optimize the mean document R-score: redact everything sensitive without touching unrelated text.

Do not train, fine-tune, calibrate or select on the validation or test split of any public dataset or benchmark; training splits and other public corpora are fine. However, you are free to use the validation data samples I have provided however you want.

Submit in /workspace/output/. Interface and runtime details: /task/CONTRACT.md. Supporting files: /task/public/. Evaluate with python /task/validate.py --server-url URL. Test the finished submission the way it will be run with frontier selfcheck. Hand it in with frontier submit when it is final. Use spin --help for compute and model APIs.

Trials

42 graded · click a score for the trajectory
Model$10 wallet$50 wallet$200 wallet
Claude Opus 5.5xhigh73.177.377.9
Claude Sonnet 5.5xhigh70.672.268.4
Claude Fable 5.1xhigh74.870.071.3
Claude Haiku 5.5max64.358.969.4
GPT-6 Astraxhigh64.868.059.0
GPT-6.1 Solxhigh60.564.355.7
GPT-6 Lunaxhigh21.818.517.4
Gemini 3.8 Flashhigh34.849.833.8
Grok 4.7xhigh43.80.031.5
Muse Spark 1.3max0.035.041.1
Muse Wondermax67.339.363.4
Kimi K3max36.356.165.5
Qwen3.8 Maxxhigh52.657.952.5
GLM 5.3max51.039.664.6

Scoring

The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.

MetricScores 0 atScores 100 at
document_r_score0.23511