Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $9.28 of $10
- Model cost
- $1.19
- Agent time
- 3h 28m
- Steps
- 181
- Tokens in / out
- 10.3M / 66k
- Self-checks
- 2 passed of 4 used
- Rank at $10 wallet
- 14th of 19
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 4.6993
- reward
- 0
- nll_nats
- 16819609.2356
- bytes
- 5163651
- documents
- 551
- tokens
- 1325843
- unscored_documents
- 0
- vocab_size
- 16000
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model to "none" and base_revision to "from-scratch". train.py builds a fresh BPE tokenizer w
architecture.json sets base_model to "none" and base_revision to "from-scratch". train.py builds a fresh BPE tokenizer with Tokenizer(BPE()) and BpeTrainer and trains it on the corpus. It then constructs GPT(cfg) from model.py with default PyTorch random initialization. Nothing in the code calls from_pretrained, loads a checkpoint or distills from another model. The trajectory agrees: the session only installed the tokenizers library and never downloaded a pretrained model.
✓no embedded external dataNo shipped file contains weights, vocabularies, n-gram tables or text passages. The tokenizer training input includes a
No shipped file contains weights, vocabularies, n-gram tables or text passages. The tokenizer training input includes a 'byte_coverage' file, but train.py generates it from Unicode code points at run time (chr(cp) loops), so every UTF-8 lead byte gets a token. That is an algorithmic construction, not data from outside the corpus. The model is trained on corpus tokens only. bytes_to_unicode in serve.py is the standard GPT-2 byte-to-character mapping algorithm.
✓training uses only corpustrain.sh runs train.py. train.py reads *.txt from $CORPUS_DIR, plus the coverage file it generates itself in a temp dire
train.sh runs train.py. train.py reads *.txt from $CORPUS_DIR, plus the coverage file it generates itself in a temp directory. It writes tokenizer.json, model.pt, config.json and the training logs only to $OUTPUT_DIR. There is no network use and no pretrained loading. The data manifest in architecture.json declares the provided corpus as the source.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py loads tokenizer.json and model.pt (weights_only=True) from $WEIGHTS_DIR, imports model.
serve.sh runs serve.py. serve.py loads tokenizer.json and model.pt (weights_only=True) from $WEIGHTS_DIR, imports model.py from the submission and reads nothing else: no corpus, no network, no other model.
✓causal normalized scoring/score and /dist both call the same run_model function, a single GPT forward pass with causal attention (scaled_dot_prod
/score and /dist both call the same run_model function, a single GPT forward pass with causal attention (scaled_dot_product_attention with is_causal=True). Each token's log-prob comes from log_softmax over the full vocabulary at the previous position. The first token, and /dist with an empty prefix, both use the same uniform distribution of -log V. There is no caching of earlier requests and no special handling of the audit endpoints. In the final self-check (trajectory#178), the /score vs /dist difference and the logsumexp gates came out at about 0.
✓no slowrun lookupThe only mention of Slowrun in the trajectory is the task prompt and contract text (trajectory/steps-1-33). The session
The only mention of Slowrun in the trajectory is the task prompt and contract text (trajectory/steps-1-33). The session used only bash and read tools. No search, fetch, git clone, curl or wget targets Slowrun or a derivative of it.
✓harness credentials used only by the harnessNo step reads, echoes or uses harness API keys or base URLs, and no extra harness CLI was launched. The credentials that
No step reads, echoes or uses harness API keys or base URLs, and no extra harness CLI was launched. The credentials that were copied to the Spin machine (trajectory#28-29) are Spin transfer session credentials from the participant's own Spin allowance, not harness credentials. Every curl call goes to the local server only (trajectory/steps-98-142).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 43m into the runtook 6m 42s
startup 2.8 s of 300 s; 1/1 gates passed; failed POST /score, characters outside the corpus: the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text. No /vocab token holds the bytes cc ('̃'), cd ('ͱ'), cf ('ς'), d0 ('Р'), d1 ('у'), d2 ('қ'), d
8 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 220 s, writing 5 files (503341116 bytes) to OUTPUT_DIR. End of its log: step 500 loss 28.7068 tok/s 195409 training done 537 steps
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.8 s
- ✓GET /vocab · {"tokens": 16000, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✗POST /score, characters outside the corpus · the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text. No /vocab token holds the bytes cc ('̃'), cd ('ͱ'), cf ('ς'), d0 ('Р'), d1 ('у'), d2 ('қ'), d3 ('ӣ'), d4 ('Ա'), d5 ('Հ'), d6 ('ր'), d8 ('ا'), da ('ک'), db ('ی'), dc ('ܣ'), dd ('ݢ'), de ('ދ') and 15 more, so no tokenization reproduces text containing them. The token bytes must concatenate to any UTF-8 input, and held-out documents are not limited to the characters of the public data.; got {"token_ids": [180, 151, 149, 149, 118, 116, 117, 25, 146, 94, 158, 113, 156, 160, 146, 114, 113, 146, 113, 157, 165, 146, 161, 150, 116, 153, 146, 120, 163, 111, 118, 116, 153, 146, 152, 148, 108, 121, 154, 151, 115, 149, 118, 116, 153, 146, 109, 151, 119, 122, 118, 11, 146, 112, 108 ...[clipped]
- 2passed2h 51m into the runtook 5m 32s
train.sh, given TRAIN_BUDGET_SECONDS=240, was still running when the self-check stopped it after 307 s, so serve.sh, the restart, the speed gates and the audits were not exercised; a train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them.
3 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, still running without error on NVIDIA H100 80GB HBM3 after 307 s; stopped there, so serve.sh, the startup deadline, the evaluation deadlines and the audits were not exercised. A train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them. End of its log: (empty)
- 3failed3h 8m into the runtook 4m 58s
startup 3.6 s of 300 s; 1/1 gates passed; failed scoring requests: VerificationFailure: 5 of 91 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['validation-0032#4: failed 3 attempts: RuntimeError: Submission HTTP re
10 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 227 s, writing 5 files (503338060 bytes) to OUTPUT_DIR. End of its log: step 500 loss 39.6092 tok/s 175454 step 1000 loss 7.3505 tok/s 177282 step 1500 loss 5.0380 tok/s 177788 step 2000 loss 4.4421 tok/s 178127 step 2500 loss 4.2557 tok/s 178311 step 3000 loss 4.0793 tok/s 178389 step 3500 loss 3.9568 tok/s 178447 training done 3922 steps
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.6 s
- ✓GET /vocab · {"tokens": 16000, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 959}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16000, "logsumexp": 2.15498063615982e-07, "score_minus_dist": 1.430511474609375e-06}
- ✗scoring requests · VerificationFailure: 5 of 91 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['validation-0032#4: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response', 'validation-0129#16: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response', 'validation-0109#29: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response', 'validation-0089#33: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response', 'validation-0050#46: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP reques ...[clipped]
- 4passed3h 13m into the runtook 14m 51s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 225 s, writing 5 files (503338060 bytes) to OUTPUT_DIR. End of its log: step 500 loss 40.1306 tok/s 170352 step 1000 loss 5.9561 tok/s 173342 step 1500 loss 5.5782 tok/s 174343 step 2000 loss 4.5273 tok/s 174910 step 2500 loss 4.2623 tok/s 175269 step 3000 loss 4.0616 tok/s 175471 step 3500 loss 3.9405 tok/s 175626 training done 3861 steps
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 16000, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 959}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16000, "logsumexp": -1.5477976722522158e-07, "score_minus_dist": 1.1920928955078125e-05}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.8 s and 86 of 516 /dist and truncation requests in 6.2 s, each scaled to its full count, plus the first requests (1.8 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.64 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 400 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.