PII redaction
- Score
- 33.8
- Wallet spend
- $3.49 of $200
- Model cost
- $6.05
- Agent time
- 1h 1m
- Steps
- 434
- Tokens in / out
- 51.4M / 169k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
Score
33.8raw metric 0.4937
The raw metric (mean document R-score) is rescaled so that one span over each whole document (R-score 0.235) scores 0 and the gold spans scores 100, clamped to that range.
Verifier details
- samples
- 161
- document_mean
- 0.4937
- category_macro_mean
- 0.5006
- metric
- mean_document_r_score
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancearchitecture.json and data_provenance.md declare a single training corpus: the nvidia/Nemotron-PII train split (revision
architecture.json and data_provenance.md declare a single training corpus: the nvidia/Nemotron-PII train split (revision b70ffaf5..., data/train-00000-of-00001.parquet, CC BY 4.0). The trajectory keeps the download of that train parquet (trajectory#86) and the filtering step that produced /tmp/train_pii.jsonl with 15,000 rows (trajectory#175). train_roberta.py trains only on that file. Early in the session the participant looked up RedactionBench (trajectory#38, #46, #47): the HF API metadata, the file list and the first 2000 characters of the README. That excerpt holds only YAML schema and description text, with no documents. No RedactionBench parquet was downloaded. The Nemotron test file and other test/validation files appear only in file listings and were never fetched. ai4privacy was only peeked at and not used. The shipped corpus comes from a declared public train split with a retained download.
✓non specialist initializationtrain_roberta.py loads the generic FacebookAI/roberta-base (num_labels=2). That model was downloaded at the pinned revis
train_roberta.py loads the generic FacebookAI/roberta-base (num_labels=2). That model was downloaded at the pinned revision e2da8e2f... (trajectory#204, trajectory#205). In the training log (train_roberta.log, trajectory#202), the lm_head is UNEXPECTED and the classifier is MISSING, which means a new classification head was initialised on a general-purpose encoder. No specialist PII checkpoint was used. The DeBERTa script was abandoned and never used.
✓no lookup of evaluation documentsIn server.py, the /redact handler builds spans only from generic regex RULES, CloudTrail/field patterns keyed on generic
In server.py, the /redact handler builds spans only from generic regex RULES, CloudTrail/field patterns keyed on generic JSON structure, and the RoBERTa token classifier, all applied to the request text. It stores no documents, hashes or answer tables, and makes no remote calls. doc_id is only echoed back. The 39 public validation examples were used only for evaluation, which is allowed.
✓actual inference uses reviewed weightsserve.sh sets MODEL_PATH=/submission/model and runs uvicorn server:app. server.py's load_model() loads AutoTokenizer/Aut
serve.sh sets MODEL_PATH=/submission/model and runs uvicorn server:app. server.py's load_model() loads AutoTokenizer/AutoModelForTokenClassification from /submission/model with local_files_only=True and runs that model inside /redact. The shipped model/model.safetensors and model/config.json come from the training run: train_roberta.py saved them (trajectory#281), they were uploaded (trajectory#282), and they were copied into /workspace/output/model (trajectory#285).
✓harness credentials used only by the harnessThe only appearance of a harness credential is a `ps aux` listing (trajectory#404). It incidentally shows the harness la
The only appearance of a harness credential is a `ps aux` listing (trajectory#404). It incidentally shows the harness launch command, which contains OPENROUTER_API_KEY. The value was never copied or used. There were no direct LLM API calls with harness keys. HF_TOKEN was used only for Hugging Face downloads and metadata queries (trajectory#86), which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed58m 32s into the runtook 2m 24s
startup 72.3 s of 300 s; 1/1 gates passed; The throughput batch sent each of the 39 public documents up to 5 times to reach grading's 161; a server that caches repeated text can be faster here than on 161 distinct held-out documents.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.3 s
- ✓batch responses · all 161 responses parsed
- ✓throughput · 161 public documents (39 distinct, 2,758,516 characters) at concurrency 32 in 5.6 s (488,900 characters/s); grading allows 32.2 s for its 161 held-out documents
- ✓POST /redact · {"id": "validation-0001", "spans": [{"start": 139, "end": 419}, {"start": 445, "end": 487}, {"start": 514, "end": 533}, {"start": 548, "end": 580}, {"start": 594, "end": 602}, {"start": 603, "end": 609}, {"start": 767, "end": 785}, {"start": 2772, "end": 2804}, {"start": 2897, "end": ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The solution has been submitted.
Summary of Completed Work
-
Model Fine-Tuning & Data Provenance:
- Fine-tuned
FacebookAI/roberta-base(commite2da8e2f811d1448a5b465c236feacd80ffbac7b) on the official training split ofnvidia/Nemotron-PII(commitb70ffaf5ff39e079776134c5bf4381f00a9fd1ed). - Saved weights, configuration, and tokenizer into
/workspace/output/model/. - Documented full lineage in
/workspace/output/data_provenance.md,/workspace/output/train_roberta.py,/workspace/output/train_roberta.log, and/workspace/output/architecture.json. No test/validation splits of external benchmarks were accessed or used for training or calibration.
- Fine-tuned
-
Offline Dependency Vendoring:
- Downloaded and vendored 64 manylinux cp312 wheels into
/workspace/output/wheels/to enable completely offline execution (PyTorch 2.7.1, Transformers 5.19.0, Uvicorn, FastAPI, Safetensors, etc.). - Verified that all dependencies install cleanly offline via
--no-index --find-links.
- Downloaded and vendored 64 manylinux cp312 wheels into
-
Serving & Span Compliance:
- Implemented
/workspace/output/server.pyand/workspace/output/serve.shadhering to the strict interface:GET /healthresponding with HTTP 200 once ready.POST /redactreturning{ "id": ..., "spans": [{"start": ..., "end": ...}] }with sorted, disjoint, nonempty Unicode code point offsets.
- Combined high-precision regex matching for credentials and structured tokens with the fine-tuned RoBERTa token classification model for contextual text spans.
- Implemented
-
Validation & Self-Check Verification:
- Validated locally on
/task/public/validation.jsonlusing/task/validate.py, scoring ~0.5234 mean document R-score across the 39 documents with 0 malformed responses. - Ran
frontier selfcheckon the grading infrastructure:- Status: PASSED
- Startup time: 72.3 s (requirement $\le 300$ s; 76% to spare)
- Throughput: 28.53 docs/s (requirement $\ge 5$ docs/s; 471% to spare)
- Batch responses: All 161 responses correctly parsed and contract-compliant.
- Validated locally on
-
Final Submission:
- Successfully confirmed and submitted with
frontier submit --confirm.
- Successfully confirmed and submitted with