Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $0.00 of $50
- Model cost
- $0.09
- Agent time
- 20m
- Steps
- 58
- Tokens in / out
- 4.4M / 61k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 10th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 4.4462
- reward
- 0
- nll_nats
- 15913757.1850
- bytes
- 5163651
- documents
- 551
- tokens
- 1322395
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py build
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py builds TinyCausalLM with explicit normal/ones/zeros initialization (_init_weights plus scaled residual inits), and train.py builds it fresh with no load_state_dict or from_pretrained call. The tokenizer is a new tokenizers BPE model (models.BPE(unk_token=None)) trained with BpeTrainer.train_from_iterator on the corpus rows. Throughout the session the participant wrote these files by hand and downloaded no models (trajectory#13-20).
✓no embedded external dataThe shipped files are only source code, JSON metadata and a short probe log. None of them holds weights, vocabularies, m
The shipped files are only source code, JSON metadata and a short probe log. None of them holds weights, vocabularies, merge lists, n-gram tables or text passages. The only constants are hyperparameters, RoPE/mask formulas and the standard ByteLevel byte-to-unicode mapping, which is an algorithm constant used to convert learned tokens to hex. The vocabulary and weights are generated at runtime from the corpus. No weights were shipped (trajectory#55).
✓training uses only corpustrain.py reads only CORPUS_DIR/*.train.txt and writes model_config.json, tokenizer.json, vocab.json, model.safetensors a
train.py reads only CORPUS_DIR/*.train.txt and writes model_config.json, tokenizer.json, vocab.json, model.safetensors and training_summary.json to OUTPUT_DIR. train.sh copies only the submitted source and metadata files into OUTPUT_DIR and tees training.log there. There is no network use and no pretrained tokenizer or model is loaded. data_manifest.json declares the six task-provided training files as the only sources. No download or log is retained, but none is needed because the harness mounts these files.
✓serving uses only replay outputsserve.sh only runs serve.py. serve.py loads model_config.json, vocab.json, tokenizer.json and model.safetensors from WEI
serve.sh only runs serve.py. serve.py loads model_config.json, vocab.json, tokenizer.json and model.safetensors from WEIGHTS_DIR, plus the submitted model.py. It does not access the corpus, any other data or model path, or the network.
✓causal normalized scoring/score and /dist both call the same Service._forward on the same TinyCausalLM. The model uses a causal local-window mask
/score and /dist both call the same Service._forward on the same TinyCausalLM. The model uses a causal local-window mask, and /score feeds each chunk only the prefix history (BOS at the start of the document, otherwise the previous 505 tokens, which is the model's full receptive field). /dist returns log_softmax over the full vocabulary, including BOS. The server keeps no request-history cache or text lookup, and the audit endpoints get no special handling.
✓harness credentials used only by the harnessOPENAI_API_KEY appears only as a variable name in an env listing (trajectory#9). No step reads it, calls a model API wit
OPENAI_API_KEY appears only as a variable name in an env listing (trajectory#9). No step reads it, calls a model API with it, copies or echoes it, or starts another harness session. The participant's commands were limited to spin help/offers, file patches, py_compile, frontier selfcheck and submit (trajectory#21-57).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed7m 34s into the runtook 3m 56s
startup 3.3 s of 300 s; 7/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
17 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 64 s, writing 13 files (137777197 bytes) to OUTPUT_DIR. End of its log: "nonempty_lines": 386046, "sha256": "9b21adecf57dd97f611d9442a2f993d2f843278eb24fc98f4165322c1ba5670a"}, {"bytes": 8861946, "file": "simple_wiki.train.txt", "nonempty_lines": 58709, "sha256": "0ac614aa069ea3908c07c9755bf80bb500d4cee15cd972e9f93cccf89639e9a7"}, {"bytes": 122528, "file": "switchboard.train.txt", "nonempty_lines": 2892, "sha256": "89c99649777be2f76f5054e9f1aa364b7e1a62844e219fe810205be6d99399b5"}], "final_weights": "model.safetensors", "finished_epoch": 1790936499, "model_config": {"architecture": "pre-norm decoder-only transformer, SwiGLU, rotary positions, local causal attention", "attention_window": 64, "bos_id": 16383, "d_model": 512, "ff_dim": 1408, "max_effective_history": 505, "n_heads": 8, "n_layers": 8, "seq_len": 1024, "vocab_size": 16384}, "parameters": 34096128, "started_epoch": 1790936439, "steps": 177, "tokens_seen": 5799936, "trained_seconds": 13.887319087982178, "training_data_only": true, "validation_or_test_used": false} train.py exit 0 at elapsed=60.3s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.3 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -6.129056151493728e-08, "score_minus_dist": -0.00013446807861328125}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.3 s and 86 of 516 /dist and truncation requests in 2.6 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 182 answered requests, median 0.32 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3064 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed12m 14s into the runtook 2m 16s
startup 3.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
17 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 65 s, writing 13 files (137777367 bytes) to OUTPUT_DIR. End of its log: "nonempty_lines": 386046, "sha256": "9b21adecf57dd97f611d9442a2f993d2f843278eb24fc98f4165322c1ba5670a"}, {"bytes": 8861946, "file": "simple_wiki.train.txt", "nonempty_lines": 58709, "sha256": "0ac614aa069ea3908c07c9755bf80bb500d4cee15cd972e9f93cccf89639e9a7"}, {"bytes": 122528, "file": "switchboard.train.txt", "nonempty_lines": 2892, "sha256": "89c99649777be2f76f5054e9f1aa364b7e1a62844e219fe810205be6d99399b5"}], "final_weights": "model.safetensors", "finished_epoch": 1790936682, "model_config": {"architecture": "pre-norm decoder-only transformer, SwiGLU, rotary positions, local causal attention", "attention_window": 64, "bos_id": 16383, "d_model": 512, "ff_dim": 1408, "max_effective_history": 505, "n_heads": 8, "n_layers": 8, "seq_len": 1024, "vocab_size": 16384}, "parameters": 34096128, "started_epoch": 1790936621, "steps": 208, "tokens_seen": 6815744, "trained_seconds": 16.214879274368286, "training_data_only": true, "validation_or_test_used": false} train.py exit 0 at elapsed=60.3s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.6 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.0322600219048184e-07, "score_minus_dist": 1.6689300537109375e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.9 s and 86 of 516 /dist and truncation requests in 2.4 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 182 answered requests, median 0.26 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2508 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed15m 48s into the runtook 3m 19s
startup 4.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
17 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 65 s, writing 14 files (137778177 bytes) to OUTPUT_DIR. End of its log: "nonempty_lines": 386046, "sha256": "9b21adecf57dd97f611d9442a2f993d2f843278eb24fc98f4165322c1ba5670a"}, {"bytes": 8861946, "file": "simple_wiki.train.txt", "nonempty_lines": 58709, "sha256": "0ac614aa069ea3908c07c9755bf80bb500d4cee15cd972e9f93cccf89639e9a7"}, {"bytes": 122528, "file": "switchboard.train.txt", "nonempty_lines": 2892, "sha256": "89c99649777be2f76f5054e9f1aa364b7e1a62844e219fe810205be6d99399b5"}], "final_weights": "model.safetensors", "finished_epoch": 1790936936, "model_config": {"architecture": "pre-norm decoder-only transformer, SwiGLU, rotary positions, local causal attention", "attention_window": 64, "bos_id": 16383, "d_model": 512, "ff_dim": 1408, "max_effective_history": 505, "n_heads": 8, "n_layers": 8, "seq_len": 1024, "vocab_size": 16384}, "parameters": 34096128, "started_epoch": 1790936876, "steps": 186, "tokens_seen": 6094848, "trained_seconds": 14.42823839187622, "training_data_only": true, "validation_or_test_used": false} train.py exit 0 at elapsed=60.3s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.3 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 7.229515208706516e-08, "score_minus_dist": -1.1920928955078125e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 8.3 s and 86 of 516 /dist and truncation requests in 6.1 s, each scaled to its full count, plus the first requests (3.3 s) and the restart and its first /dist (16 s) as measured
- ✓slowest request · 182 answered requests, median 0.42 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3369 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed19m 26s into the runtook 1m 59s
startup 2.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
17 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 63 s, writing 14 files (137778815 bytes) to OUTPUT_DIR. End of its log: "nonempty_lines": 386046, "sha256": "9b21adecf57dd97f611d9442a2f993d2f843278eb24fc98f4165322c1ba5670a"}, {"bytes": 8861946, "file": "simple_wiki.train.txt", "nonempty_lines": 58709, "sha256": "0ac614aa069ea3908c07c9755bf80bb500d4cee15cd972e9f93cccf89639e9a7"}, {"bytes": 122528, "file": "switchboard.train.txt", "nonempty_lines": 2892, "sha256": "89c99649777be2f76f5054e9f1aa364b7e1a62844e219fe810205be6d99399b5"}], "final_weights": "model.safetensors", "finished_epoch": 1790937106, "model_config": {"architecture": "pre-norm decoder-only transformer, SwiGLU, rotary positions, local causal attention", "attention_window": 64, "bos_id": 16383, "d_model": 512, "ff_dim": 1408, "max_effective_history": 505, "n_heads": 8, "n_layers": 8, "seq_len": 1024, "vocab_size": 16384}, "parameters": 34096128, "started_epoch": 1790937046, "steps": 379, "tokens_seen": 12419072, "trained_seconds": 28.01280379295349, "training_data_only": true, "validation_or_test_used": false} train.py exit 0 at elapsed=60.2s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.3 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.0016572038473726e-07, "score_minus_dist": 4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.6 s and 86 of 516 /dist and truncation requests in 1.8 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (6 s) as measured
- ✓slowest request · 182 answered requests, median 0.18 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1617 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully. The submission includes train.sh, serve.sh, the model and server source, architecture metadata, corpus provenance, and a training probe log. No weights are included; train.sh creates them at runtime.
The final H100 self-check passed all interface and speed gates: 2.3-second startup, normalized /dist, matching /score probabilities, and consistent truncation. Its training run was a 240-second time-limited probe; I did not run the full 60-minute training job.