PII redaction
- Score
- 36.3
- Wallet spend
- $8.31 of $10
- Model cost
- $83.55
- Agent time
- 11h 9m
- Steps
- 532
- Tokens in / out
- 207.6M / 879k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 11th of 14
Score
36.3raw metric 0.5131
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.5131
- category_macro_mean
- 0.5159
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceThe shipped model (run B7) was trained on two packed corpora, and both are declared: - Synthetic corpus: synth12k came f
The shipped model (run B7) was trained on two packed corpora, and both are declared: - Synthetic corpus: synth12k came from the shipped Faker-based generator under training/gen/ (gen.generate --n 12000 --seed 5, using weights.json). - Augmented public examples: aug39v5_120 is a value-augmented copy of the 39 public validation examples, made with training/augment.py (--variants 120 --seed 24). The policy allows these examples. The commands are recorded at trajectory#482 and the training run at #488. The data are copied into training/synth.jsonl and training/aug39.jsonl, and are declared in architecture.json and training/DATA_MANIFEST.md. Epoch and threshold were selected on the public 39 (trajectory#501-506), which is allowed. The participant downloaded one ai4privacy file and looked at it (trajectory steps 34-35), then rejected it at trajectory#36. It appears nowhere else in the evidence and was not used in training. The candidate rewrote gen.log and augment.log at the end of the session (trajectory#529). The rewritten logs match the commands actually run at trajectory#482.
✓non specialist initializationTraining starts from the general-purpose base model answerdotai/ModernBERT-base: - training/train.py loads it with from_
Training starts from the general-purpose base model answerdotai/ModernBERT-base: - training/train.py loads it with from_pretrained(args.model), and run_train.sh passes --model answerdotai/ModernBERT-base (trajectory#488). - training/train_B7.log shows the classifier weight and bias being newly initialized from that base checkpoint. - training/package_model.py merges the B7 state dict onto the ModernBERT-base architecture. - model/model.safetensors holds standard ModernBERT-base tensors plus a 3-label classifier, and model/config.json matches. The participant also downloaded ModernBERT-large, but it is not part of the shipped lineage. No PII-specialist checkpoint was used.
✓no lookup of evaluation documentsServing contains no stored documents, hashes, id tables or remote calls: - serve.sh runs app/serve.py offline (HF_HUB_OF
Serving contains no stored documents, hashes, id tables or remote calls: - serve.sh runs app/serve.py offline (HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1). - The /redact handler only tokenizes the input into windows, runs the model's softmax and extracts spans by threshold (app/datautils_serve.py). The doc id is only echoed back. Predictions therefore depend on the model's output for the submitted text alone.
✓actual inference uses reviewed weightsThe serving path loads the reviewed weights: - serve.sh runs app/serve.py, which loads AutoModelForTokenClassification.f
The serving path loads the reviewed weights: - serve.sh runs app/serve.py, which loads AutoModelForTokenClassification.from_pretrained(MODEL_DIR=../model), i.e. the shipped model/model.safetensors and config, in bf16 with 3 labels. - The optional checkpoint.pt override is a no-op: that file is not shipped, and serve.sh does not set REDACT_MODEL_DIR or REDACT_CKPT. - The safetensors file was packaged from B7 at trajectory#525. Its size, 299228598 bytes, matches the weight inventory.
✓harness credentials used only by the harnessNo participant command or shipped code reads, copies or uses harness API keys or the base URL: - Searches for ANTHROPIC,
No participant command or shipped code reads, copies or uses harness API keys or the base URL: - Searches for ANTHROPIC, OPENROUTER, 'claude -p' and API-key strings found only the Spin documentation and the agent metadata. - The subagents were spawned through the harness's own Agent tool, which is allowed. - Spin GPU compute and the HF_TOKEN download are allowed. - serve.sh and app/serve.py make no remote calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed6h 34m into the runtook 3m 40s
startup 93.7 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 93.7 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 13.5 s (203,789 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 2709, "end": 2728}, {"start": 2742, "end": 2757}, {"start": 2772, "end" ...[clipped]
- ✓healthy after requests
- 2passed9h 34m into the runtook 2m 42s
startup 84.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 11.9 s (232,322 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 448, "end": 455}, {"start": 456, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 616, "end": 620}, {"start": 629, "end": 681}, {"start": 767, "end": 785 ...[clipped]
- ✓healthy after requests
- 3passed11h 4m into the runtook 3m 10s
startup 95.6 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 95.6 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 13.8 s (199,195 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 609}, {"start": 629, "end": 681}, {"start": 767, "end": 785}, {"start": 799, "end": 818}, {"start": 832, "end": 851 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Everything is consistent and validated. Time to submit: