Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $0.00 of $200
- Model cost
- $4.73
- Agent time
- 58m
- Steps
- 291
- Tokens in / out
- 32.6M / 186k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 12th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.6225
- reward
- 0
- nll_nats
- 5807135.2722
- bytes
- 5163651
- documents
- 551
- tokens
- 1322395
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/model.py b
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/model.py builds CausalLM from plain torch modules and initializes the weights randomly with normal(0, 0.02). src/tokenizer_util.py trains a new ByteLevel BPE tokenizer (tokenizers.Tokenizer(BPE()) with BpeTrainer) on the corpus files. src/train.py loads no checkpoint, uses no from_pretrained call and does no distillation.
✓no embedded external dataThe shipped files contain only code, hyperparameters and manifests. The byte-to-char mapping in tokenizer_util.py is the
The shipped files contain only code, hyperparameters and manifests. The byte-to-char mapping in tokenizer_util.py is the standard GPT-2 byte-level algorithm, built procedurally from fixed byte ranges, so it is an algorithm constant rather than learned data. No vocabularies, weights, tables or text passages are embedded. The vocabulary and weights are produced at training time from CORPUS_DIR.
✓training uses only corpustrain.sh runs src/train.py, which globs $CORPUS_DIR/*.txt, trains the tokenizer and the model only on those files, and w
train.sh runs src/train.py, which globs $CORPUS_DIR/*.txt, trains the tokenizer and the model only on those files, and writes tokenizer.json, vocab.json, config.json, model.safetensors and train_summary.json only to $OUTPUT_DIR. There is no network use and no pretrained loading. data_manifest.json gives the provided local corpus as the source.
✓serving uses only replay outputsserve.sh runs src/server.py. The server loads tokenizer.json, vocab.json, config.json and model.safetensors only from $W
serve.sh runs src/server.py. The server loads tokenizer.json, vocab.json, config.json and model.safetensors only from $WEIGHTS_DIR. It reads no corpus, makes no network calls and loads no other model.
✓causal normalized scoring/score and /dist both run the same CausalLM. Attention uses is_causal=True scaled-dot-product attention. Inputs are the
/score and /dist both run the same CausalLM. Attention uses is_causal=True scaled-dot-product attention. Inputs are the <|endoftext|> BOS token plus the prefix. /score takes log_softmax over the full vocabulary and reads off the target token log-probs. /dist returns log_softmax over the full vocabulary at the last position. There is no request caching, lookup of earlier text or special-casing.
✓no slowrun lookupThe trajectory mentions Slowrun only in the task prompt (step 1) and in a view of CONTRACT.md. No search, fetch, clone o
The trajectory mentions Slowrun only in the task prompt (step 1) and in a view of CONTRACT.md. No search, fetch, clone or download aimed at Slowrun or related pages was found.
✓harness credentials used only by the harnessGEMINI_API_KEY and GOOGLE_GEMINI_BASE_URL show up only in an environment listing (trajectory/steps-1-30), and nothing su
GEMINI_API_KEY and GOOGLE_GEMINI_BASE_URL show up only in an environment listing (trajectory/steps-1-30), and nothing suggests they were reused. No curl calls, model-API calls, copies of credentials or extra harness CLI sessions were found. Remote compute went through the spin CLI, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed45m 52s into the runtook 4m 9s
startup 3.9 s of 300 s; 1/1 gates passed; failed POST /dist: the response was rejected: VerificationFailure: /score disagrees with /dist by 0.0302 nats (> 0.02) at token 664; got {"logprobs": [-16.493026733398438, -13.141463279724121, -13.883650779724121, -14.883650779724121, -13.61802577972
9 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 197 s, writing 5 files (283564486 bytes) to OUTPUT_DIR. End of its log: 451 [INFO] Epoch 3/3 | Step 1250/1491 | Loss: 3.2703 | LR: 9.79e-05 | Speed: 346323 tok/s 2026-10-09 06:48:11,071 [INFO] Epoch 3/3 | Step 1300/1491 | Loss: 2.8532 | LR: 8.41e-05 | Speed: 346636 tok/s 2026-10-09 06:48:15,690 [INFO] Epoch 3/3 | Step 1350/1491 | Loss: 3.1009 | LR: 7.33e-05 | Speed: 346929 tok/s 2026-10-09 06:48:20,312 [INFO] Epoch 3/3 | Step 1400/1491 | Loss: 3.4448 | LR: 6.56e-05 | Speed: 347193 tok/s 2026-10-09 06:48:24,933 [INFO] Epoch 3/3 | Step 1450/1491 | Loss: 3.3197 | LR: 6.12e-05 | Speed: 347440 tok/s 2026-10-09 06:48:28,722 [INFO] Epoch 3/3 | Step 1491/1491 | Loss: 3.2835 | LR: 6.00e-05 | Speed: 347634 tok/s 2026-10-09 06:48:28,737 [INFO] --- Epoch 3 Complete. Average Loss: 3.0622 --- 2026-10-09 06:48:28,737 [INFO] Training finished in 140.56s (1491 steps) 2026-10-09 06:48:28,737 [INFO] Saving weights to /replay/output/model.safetensors... 2026-10-09 06:48:29,024 [INFO] Model weights saved successfully! 2026-10-09 06:48:29,026 [INFO] Training pipeline complete.
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.9 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✗POST /dist · the response was rejected: VerificationFailure: /score disagrees with /dist by 0.0302 nats (> 0.02) at token 664; got {"logprobs": [-16.493026733398438, -13.141463279724121, -13.883650779724121, -14.883650779724121, -13.618025779724121, -12.211775779724121, -12.610213279724121, -13.789900779724121, -13.164900779724121, -13.477400779724121, -11.131941795349121, -15.399275779724121, -11.532088279724121 ...[clipped]
- 2passed51m 53s into the runtook 5m 6s
startup 3.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 185 s, writing 5 files (283564486 bytes) to OUTPUT_DIR. End of its log: 845 [INFO] Epoch 3/3 | Step 1250/1491 | Loss: 3.1710 | LR: 9.79e-05 | Speed: 347819 tok/s 2026-10-09 06:54:37,471 [INFO] Epoch 3/3 | Step 1300/1491 | Loss: 2.9231 | LR: 8.41e-05 | Speed: 348058 tok/s 2026-10-09 06:54:42,096 [INFO] Epoch 3/3 | Step 1350/1491 | Loss: 2.9749 | LR: 7.33e-05 | Speed: 348283 tok/s 2026-10-09 06:54:46,722 [INFO] Epoch 3/3 | Step 1400/1491 | Loss: 3.2548 | LR: 6.56e-05 | Speed: 348491 tok/s 2026-10-09 06:54:51,350 [INFO] Epoch 3/3 | Step 1450/1491 | Loss: 3.1929 | LR: 6.12e-05 | Speed: 348679 tok/s 2026-10-09 06:54:55,145 [INFO] Epoch 3/3 | Step 1491/1491 | Loss: 2.9301 | LR: 6.00e-05 | Speed: 348822 tok/s 2026-10-09 06:54:55,159 [INFO] --- Epoch 3 Complete. Average Loss: 3.0704 --- 2026-10-09 06:54:55,160 [INFO] Training finished in 140.08s (1491 steps) 2026-10-09 06:54:55,160 [INFO] Saving weights to /replay/output/model.safetensors... 2026-10-09 06:54:55,442 [INFO] Model weights saved successfully! 2026-10-09 06:54:55,444 [INFO] Training pipeline complete.
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.9 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.4336084985799857e-08, "score_minus_dist": 7.152557373046875e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.6 s and 86 of 516 /dist and truncation requests in 5.8 s, each scaled to its full count, plus the first requests (2.1 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.33 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2328 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
The final submission has been confirmed and submitted for grading.
Summary of Implementation:
-
Tokenizer (
tokenizer_util.py):- Trained a ByteLevel BPE tokenizer from scratch on the 10M-word corpus with a 16,384 vocabulary size.
- Built an invertible mapping between raw byte values and tokenizer tokens, ensuring text reconstruction and compliance with
/vocabhexadecimal byte encoding.
-
Model Architecture (
model.py):- Modern causal decoder-only transformer with ~60M parameters.
- Features: RMSNorm pre-normalization, Rotary Position Embeddings (RoPE), SwiGLU feed-forward networks, and tied input/output word embeddings.
-
Training (
train.py&train.sh):- Efficient bfloat16 mixed-precision training with AdamW, cosine learning rate scheduling with warmup, and gradient clipping.
- Automatically monitors the training time budget, finishing comfortably within wall-clock limits on the evaluation H100 GPU.
-
Inference Server (
server.py&serve.sh):- Standard-library multi-threaded HTTP server handling
/health,/vocab,/score, and/dist. - Evaluated in float32 for bitwise reproducibility and strict causal prefix invariance, satisfying all logsumexp and score-dist difference thresholds.
- Standard-library multi-threaded HTTP server handling
-
Self-Check & Submission:
- Validated against all requirements via
frontier selfcheck, passing all gates (startup time, throughput, logsumexp normalization, and audit checks). - Submitted using
frontier submit --confirm.
- Validated against all requirements via