Data-efficient pretraining
- Score
- 48.2
- Wallet spend
- $8.73 of $10
- Model cost
- $5.77
- Agent time
- 1h 59m
- Steps
- 178
- Tokens in / out
- 15.8M / 135k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 6th of 19
Score
48.2raw metric 0.4823
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3592
- reward
- 0.4823
- nll_nats
- 4864950.1569
- bytes
- 5163651
- documents
- 551
- tokens
- 1255996
- unscored_documents
- 0
- vocab_size
- 32768
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. data.py trains a ne
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. data.py trains a new ByteLevel BPE with tokenizers' BpeTrainer.train_from_iterator, using only the corpus training chunks. model.py builds CausalLM with random normal initialization (std 0.02) and no from_pretrained or checkpoint load. train.py trains it with AdamW and does not load any pretrained state. No pretrained model or tokenizer appears anywhere in the code or the trajectory.
✓no embedded external dataThe shipped source holds only algorithm code and hyperparameters. The only constants are FNV hash constants, the byte-le
The shipped source holds only algorithm code and hyperparameters. The only constants are FNV hash constants, the byte-level alphabet ranges and seeds. The vocab, merges, n-gram tables and weights are all computed at replay time from CORPUS_DIR (data.py, ngram.py fit, train.py, combine.py). No embedded tables, text or encoded data were found.
✓training uses only corpustrain.sh runs train.py and combine.py with --corpus-dir $CORPUS_DIR and output under $OUTPUT_DIR. prepare_corpus reads o
train.sh runs train.py and combine.py with --corpus-dir $CORPUS_DIR and output under $OUTPUT_DIR. prepare_corpus reads only CORPUS_DIR/*.train.txt and carves its selection proxy out of those same files. All outputs go to OUTPUT_DIR or its models/ subdirectories. There are no network calls and no external model loads. data_manifest.json declares the task-supplied corpus as the only source. During development the participant moved the task corpus to the GPU machine through Spin storage (trajectory#36); that was a transfer of the same files, not new data.
✓serving uses only replay outputsserve.sh only runs server.py. ProbabilityEngine reads tokenizer.json, vocab.json, blend.json, ensemble.json, config/mode
serve.sh only runs server.py. ProbabilityEngine reads tokenizer.json, vocab.json, blend.json, ensemble.json, config/model.safetensors per model and ngram.npz, all under WEIGHTS_DIR. ensemble.json holds relative paths to models/model_i under the output directory, which train.sh writes. There is no corpus, network or other path access.
✓causal normalized scoring/score and /dist use the same models: a uniform ensemble of causal transformers (SDPA with is_causal=True) and the same
/score and /dist use the same models: a uniform ensemble of causal transformers (SDPA with is_causal=True) and the same corpus n-gram, mixed with the same weight. score_neural and dist_neural both pick context through the same _window_for_target, so each log-prob depends only on its prefix. /dist applies log_softmax over the full vocabulary, mixes it with the n-gram's normalized distribution, then renormalizes. The n-gram's probabilities and distribution use the same backoff with prefix-only contexts. There is no request caching, history or endpoint special-casing.
✓no slowrun lookupThe only Slowrun or qlabs mentions in the trajectory are the task prompt and the CONTRACT text. There are no WebSearch o
The only Slowrun or qlabs mentions in the trajectory are the task prompt and the CONTRACT text. There are no WebSearch or WebFetch calls; the tool calls are only Bash, Read, Edit and Write. No git clone, wget or curl targets Slowrun. The only curl downloads fetched the s5cmd release from GitHub (trajectory#33), and a local health check hit the participant's own server on 127.0.0.1.
✓harness credentials used only by the harnessSearches for ANTHROPIC, API_KEY, BASE_URL, printenv and harness CLI launches find no use of harness credentials. The one
Searches for ANTHROPIC, API_KEY, BASE_URL, printenv and harness CLI launches find no use of harness credentials. The one ANTHROPIC/OPENAI hit is a scan of the output files for leaked strings (trajectory#139-160 chunk). The S3 credentials came from a Spin storage session and were used only to move the corpus (trajectory#34-36). Those are Spin credentials, not harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 6m into the runtook 38s
failed train.sh: With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: train start=1792806846 replay_start=1792806846 budget=240 corpus=/corpus output=/replay/output seeds=1 insufficient time for model 0; preserving completed models no model completed successfully
3 gates
- ✓review prerequisites
- ✓entry points and files
- ✗train.sh · With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: train start=1792806846 replay_start=1792806846 budget=240 corpus=/corpus output=/replay/output seeds=1 insufficient time for model 0; preserving completed models no model completed successfully
- 2passed1h 7m into the runtook 4m 21s
startup 6.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 142 s, writing 20 files (3097266412 bytes) to OUTPUT_DIR. End of its log: 3, 'simple_wiki': 1.7053079476387178, 'switchboard': 1.3741823221608225} [ 108.0s] selected raw@1.000/step831 using only the training-corpus proxy [ 108.3s] finished in 108.3s; final proxy bpb=1.490682 combining 1 completed models ensemble subsets: [{'models': [0], 'bpb': 1.490682419452936, 'nll': 153265.86587976705}] ngram: 15,693,975 target tokens; fitting orders 2..10 ngram order 2: 32,444 contexts, 1,808,859 pairs ngram order 3: 1,808,840 contexts, 5,460,443 pairs ngram order 4: 5,460,001 contexts, 8,571,535 pairs ngram order 5: 8,569,910 contexts, 10,519,589 pairs ngram order 6: 10,516,883 contexts, 11,681,077 pairs ngram order 7: 11,677,721 contexts, 12,434,104 pairs ngram order 8: 12,430,476 contexts, 13,195,477 pairs ngram order 9: 13,191,731 contexts, 13,823,353 pairs ngram order 10: 13,819,526 contexts, 14,225,544 pairs selected models=(0,) blend={'weight': 0.3, 'discount': 0.65, 'alpha': 0.1, 'nll': 139015.6341346912, 'bpb': 1.352082935389218} train complete elapsed=141
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.0 s
- ✓GET /vocab · {"tokens": 32768, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1226}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32768, "logsumexp": -2.220446049250313e-16, "score_minus_dist": -2.6695340848981175e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.2 s and 86 of 516 /dist and truncation requests in 5.8 s, each scaled to its full count, plus the first requests (2.2 s) and the restart and its first /dist (12 s) as measured
- ✓slowest request · 183 answered requests, median 0.39 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4821 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed1h 55m into the runtook 4m 10s
startup 6.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 147 s, writing 20 files (3097266416 bytes) to OUTPUT_DIR. End of its log: 11, 'simple_wiki': 1.772138178319465, 'switchboard': 1.41940544362324} [ 108.2s] selected raw@1.000/step717 using only the training-corpus proxy [ 108.5s] finished in 108.5s; final proxy bpb=1.531329 combining 1 completed models ensemble subsets: [{'models': [0], 'bpb': 1.531328969818027, 'nll': 157444.97784582718}] ngram: 15,693,975 target tokens; fitting orders 2..10 ngram order 2: 32,444 contexts, 1,808,859 pairs ngram order 3: 1,808,840 contexts, 5,460,443 pairs ngram order 4: 5,460,001 contexts, 8,571,535 pairs ngram order 5: 8,569,910 contexts, 10,519,589 pairs ngram order 6: 10,516,883 contexts, 11,681,077 pairs ngram order 7: 11,677,721 contexts, 12,434,104 pairs ngram order 8: 12,430,476 contexts, 13,195,477 pairs ngram order 9: 13,191,731 contexts, 13,823,353 pairs ngram order 10: 13,819,526 contexts, 14,225,544 pairs selected models=(0,) blend={'weight': 0.3, 'discount': 0.65, 'alpha': 0.1, 'nll': 141497.66924623444, 'bpb': 1.3762235102263114} train complete elapsed=147
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.6 s
- ✓GET /vocab · {"tokens": 32768, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1226}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32768, "logsumexp": -2.220446049250313e-16, "score_minus_dist": 4.565138868928287e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.4 s and 86 of 516 /dist and truncation requests in 5.8 s, each scaled to its full count, plus the first requests (2.2 s) and the restart and its first /dist (14 s) as measured
- ✓slowest request · 183 answered requests, median 0.39 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2064 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
The exact frozen submission passed the final self-check: 8/8 gates, digest 12e190b1e46eeadf50605b59663dc1c96edf49461bb8a0ff0f16768316ea979f. I’m handing it in now.