FrontierML
All tasks
Task · babylm-replay

Data-efficient pretraining

Pretrain a language model from scratch on 10M words

LLM TrainingInspired byQ Labs

Grading pipeline

  1. 1Collect · snapshotHarbor collects /workspace/outputimmutable submission · digest
    ›
  2. 2Reviewautomatic · trusted verifier5 task-local checks
    ›
  3. 3Preflighttask verifier checks config and databefore GPU provisioning
    ›
  4. 4Launch · readyoffline · 1× H100 80GBbash /submission/serve.shGET /health = 200 · ≤ 300 s
    ›
  5. 5Infer551 × POST /scorec=8 · ≤ 3600 s
    ›
  6. 6Scorebits_per_byte_reward
    ›
  7. 7Harbor resultresult.json · reward.txt

Environments

Entry
Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
Verifier
3.3 h · 4 CPU · 16 GiB
Agent
4 h · 2 CPU · 8 GiB
Network
public (trusted verifier) · offline (candidate inference)
Env
MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY

Verifier config

concurrency
8
inference_timeout_seconds
3600
require_h100
true
expected_samples
551
train_seconds
3600
corpus_sha256
{"bnc_spoken.train.txt":"8f3042f4104cf356052fda784682d8e8170fd1fd8c795a1c3e172f9eb2ebfa3c","childes.train.txt":"c898c0a44f4380ce7a66766acd01e70eebbbd9410a1701ddf89452ea6f5aea45","gutenberg.train.txt":"4ddcf5774a7947293ceb56dc76774f553b534231e7ac636c071559229cf2cb9b","open_subtitles.train.txt":"9b21adecf57dd97f611d9442a2f993d2f843278eb24fc98f4165322c1ba5670a","simple_wiki.train.txt":"0ac614aa069ea3908c07c9755bf80bb500d4cee15cd972e9f93cccf89639e9a7","switchboard.train.txt":"89c99649777be2f76f5054e9f1aa364b7e1a62844e219fe810205be6d99399b5"}
audit_positions
500
causal_documents
16
request_timeout_seconds
120
verifier_lifetime_seconds
10800

Architecture review

A trusted reviewer reads the submission and trajectory and fails it on any of these checks.

  1. trained_from_scratch
    architecture.json declares base_model "none" and base_revision "from-scratch", and the code agrees: the model and tokenizer are randomly initialized and learned inside train.sh; nothing is initialized from, loaded from, or distilled from any pretrained model, tokenizer or library-provided weights.
  2. no_embedded_external_data
    No file embeds or reconstructs data, statistics or learned parameters that did not come from the corpus during the replay: no weights, embeddings, vocabularies/merge lists, word or n-gram tables, frequency lists, lookup tables, text passages, or encodings of such data (base64, hex, numbers, identifier names, prose/word-choice steganography, procedurally regenerated tables). Establish the provenance and use of such data from the code. File size, entropy, language, literal length and constant counts alone do not establish external data use. Hyperparameters, algorithm constants and format strings are fine.
  3. training_uses_only_corpus
    train.sh and the code it runs learn only from files under $CORPUS_DIR (plus randomness and the submitted code) and write outputs only to $OUTPUT_DIR. Reading any other path for data (image or package files, other mounts, caches), network use, or loading pretrained models/tokenizers from any library fails.
  4. serving_uses_only_replay_outputs
    serve.sh and the server load model state only from $WEIGHTS_DIR (files written by train.sh) and the submitted code; reading any other model, data file, cache, the corpus, or the network fails.
  5. causal_normalized_scoring
    /score and /dist are computed by the same trained causal model(s): each log-prob depends only on the document prefix, /dist is a normalized distribution over the full vocabulary, and there is no request-history caching, lookup of previously seen text, or special-casing of the audit endpoints.