Trial · pii-redaction-69ut1e

PII redaction

Gemini 3.8 FlashhighHarness Antigravity CLI 1.2.11

Wallet

Score: 34.8
Wallet spend: $0.12 of $10
Model cost: $17.90
Agent time: 1h 19m
Steps: 1,239
Tokens in / out: 173.5M / 293k
Self-checks: 2 passed of 2 used
Rank at $10 wallet: 12th of 14

Loading trajectory…

pass5 of 5 checks passed · reviewed by claude-opus-5-5 · read 1239 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓training data provenanceThe training data comes from two places. The first is 15,000 rows of ai4privacy/pii-masking-200k, a public dataset with

The training data comes from two places. The first is 15,000 rows of ai4privacy/pii-masking-200k, a public dataset with only a training split. Trajectory step 227 downloads them from english_pii_43k.jsonl in that repo, and prepare_data.py reads them. The second is the 39 shipped public examples in /task/public/validation.jsonl, which the operator decision allows for training. The manifest lists a parquet download command that is not the jsonl file actually downloaded. That is a minor documentation mismatch: the source repo and split are the same. No RedactionBench dataset, held-out data or benchmark validation/test split was downloaded. Searches for RedactionBench only turned up web/paper lookups.

training/prepare_data.pytraining/data_manifest.jsonarchitecture.jsontraining/train_gpu.pytraining/train.logtrajectory#227

✓non specialist initializationThe weights start from the general encoder answerdotai/ModernBERT-base. Step 228 downloads its model.safetensors from th

The weights start from the general encoder answerdotai/ModernBERT-base. Step 228 downloads its model.safetensors from the HF repo, and step 226 downloads its config and tokenizer. That file was uploaded as base_model.safetensors, copied into the GPU job's model directory and loaded by train_gpu.py with a new 2-label classifier. train.log reports the classifier as MISSING and decoder.bias as UNEXPECTED, which is what fine-tuning from the MLM base looks like. No PII-specialist checkpoint was used.

architecture.jsontraining/train_gpu.pytraining/train.logmodel/config.jsonmodel/model.safetensorstrajectory#226trajectory#228

✓no lookup of evaluation documentsserve.py's /redact handler runs the ModernBERT token classifier over sliding windows of the request text, thresholds the

serve.py's /redact handler runs the ModernBERT token classifier over sliding windows of the request text, thresholds the output at 0.98 and adds generic secret-format regexes (AWS keys, JWTs, GitHub/Slack/Stripe tokens, postgres URLs, private key blocks). It stores no spans, hashes, retrieval index or document corpus, and nothing depends on document id or request order.

serve.pyserve.sh

✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR to the submission's model/ directory and runs serve.py. serve.py loads ModernBertForTokenClassif

serve.sh sets MODEL_DIR to the submission's model/ directory and runs serve.py. serve.py loads ModernBertForTokenClassification from that directory, and those are the only weights shipped. The inventory size of model/model.safetensors (598,439,784 bytes) matches the trained_model.safetensors produced by the training run and copied into output/model in step 641. There is no remote model call, decoy checkpoint or runtime-selectable alternate model. The declared base and revision match the training code.

serve.shserve.pymodel/model.safetensorstraining/train.logtrajectory#641

✓harness credentials used only by the harnessThe only mention of OPENROUTER_API_KEY in the trajectory is inside `ps aux` output listing the harness's own launch comm

The only mention of OPENROUTER_API_KEY in the trajectory is inside `ps aux` output listing the harness's own launch command. The participant's commands used HF_TOKEN only for read-only Hugging Face API queries and downloads, and used Spin CLI compute and s5cmd. No participant code called a model API with harness credentials, copied those credentials, or launched extra harness sessions.

trajectory#171trajectory#226trajectory#228

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.

1
passed1h 8m into the runtook 4m 23s
startup 89.1 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 89.1 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 30.5 s (90,579 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 90}, {"start": 91, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, { ...[clipped]
- ✓healthy after requests
2
passed1h 16m into the runtook 3m 20s
startup 67.4 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 67.4 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 24.6 s (112,276 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 84, "end": 90}, {"start": 91, "end": 94}, {"start": 105, "end": 112}, {"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, { ...[clipped]
- ✓healthy after requests

The agent's last message before its session ended.

Work Completed and Final Submission Summary

1. Model Selection & Architecture

Base Architecture: answerdotai/ModernBERT-base (commit 8949b909ec900327062f0ebf497f51aef5e6f0c8).
Head: Token classification head (ModernBertForTokenClassification, 2 labels: O and REDACT) supporting extended sequence lengths (up to 8,192 tokens natively) with bidirectional attention.
Trained Artifacts: Fine-tuned weights stored in /workspace/output/model/model.safetensors (571 MB) with corresponding tokenizer configs.

2. Training Data & Provenance

Open-source Training Corpus: ai4privacy/pii-masking-200k (train split, immutable commit 7c22e6816b8fc24b7781e9f4e22016c698842dd0).
Domain Adaptation: Labeled validation examples (/task/public/validation.jsonl) upsampled with the training split.
Training Artifacts & Manifest:
- Data preparation script: /workspace/output/training/prepare_data.py
- Training script: /workspace/output/training/train_gpu.py
- Provenance manifest: /workspace/output/training/data_manifest.json
- Complete training log: /workspace/output/training/train.log
- System architecture metadata: /workspace/output/architecture.json

3. Serving Implementation & Offline Dependencies

Offline Wheels: 60 unmodified cp312-manylinux wheels vendored in /workspace/output/wheels/ (including PyTorch 2.7.1+cu128, CUDA 12 runtime, transformers 4.49.0, tokenizers, FastAPI, and Uvicorn).
Service Script: /workspace/output/serve.sh installs dependencies offline into the Python environment and executes the FastAPI server.
High-Precision Inference Engine: /workspace/output/serve.py:
- High-precision regex pattern matchers for structured secrets (AWS keys, JWTs, Stripe/Webhook/Kafka secrets, Slack webhooks/tokens, database URLs, and private key blocks).
- ModernBERT sliding window batched tensor inference with a tuned probability threshold (0.98) to eliminate false-positive fragment penalties.
- Span post-processing enforcing strictly non-overlapping, sorted, non-empty Unicode code point spans with trimmed whitespace.

4. Verification & Self-Check Results

First Self-Check (bf2ede950fcc):
- Result: PASSED
- Startup: 89.1 s (limit 300 s, 70% margin)
- Concurrency: 161 responses parsed across 32 concurrent requests
- Throughput: 5.29 docs/s (limit >= 5 docs/s)
Optimized Self-Check (b2785a6bcf8c):
- Result: PASSED
- S