Trial · llm-post-train-3ork9d

LLM post-training

GLM 5.3maxHarness Claude Code 2.1.291

Wallet

Score: 41.2
Wallet spend: $42.67 of $50
Model cost: $7.56
Agent time: 10h 52m
Steps: 318
Tokens in / out: 31.3M / 447k
Self-checks: 1 passed of 1 used
Rank at $50 wallet: 6th of 14

Loading trajectory…

Score

41.2raw metric 0.4946

The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.

Benchmarks

Benchmark	Raw	Floor	Ceiling	Normalized
gpqa_diamond	0.409	0.250	0.596	46.0
math_500	0.716	0.622	0.956	28.1
livecodebench_v6	0.283	0.200	0.490	28.8
mmlu_pro	0.570	0.361	0.699	61.8

Verifier details

metric: four_benchmark_mean
suite: gpqa-math500-lcbv6-mmlupro-v6
harmonic_mean: 0.4384

pass4 of 4 checks passed · reviewed by claude-opus-5-5 · read 318 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓pinned base startThe GPU box downloaded the base with snapshot_download("Qwen/Qwen3-4B-Base", revision="906bfd4b4dc7f14ee4320094d8b41684a

The GPU box downloaded the base with snapshot_download("Qwen/Qwen3-4B-Base", revision="906bfd4b4dc7f14ee4320094d8b41684abff8539", local_dir="/root/work/qwen3-4b-base") (trajectory#60–66), which matches the pinned revision. In code/train.py, --base defaults to that directory and is loaded with AutoModelForCausalLM.from_pretrained(args.base), so the main SFT started from the pinned base. Remedial pass 1 loaded /root/work/runs/sft/final (training/remedial_stats.json, trajectory#256). Remedial pass 2 loaded /root/work/runs/remedial/final (training/remedial2_stats.json, trajectory#297–298). The final runs/remedial2/final/model.safetensors was uploaded and downloaded with a matching sha256, copied to model/model.safetensors, and checked against the base as 398 tensors with matching names and shapes in BF16 (trajectory#313–314). The trusted checkpoint audit agrees: same parameter count as the base and tied head. The trajectory shows no other model being loaded.

code/train.pytraining/remedial_stats.jsontraining/remedial2_stats.jsontraining/remedial2_log.txtarchitecture.jsonmodel/model.safetensorstrajectory#60trajectory#256trajectory#313trajectory#314

✓no public benchmark eval trainingAll training sources are declared: - OpenR1-Math-220k train, nvidia/OpenCodeReasoning split_0 and smoltalk (code/prepare

All training sources are declared: - OpenR1-Math-220k train, nvidia/OpenCodeReasoning split_0 and smoltalk (code/prepare_data.py). - MCQ prompts from the train splits of ARC, OpenBookQA, SciQ and MMLU auxiliary_train (code/gen_mcq.py), with teacher completions from the participant's own spin key. GPQA and LCB question texts were used only for 10-gram decontamination. Copying the GPQA prompt format is not a use of benchmark items. Both remedial datasets are length-filtered subsets of the same train.jsonl (trajectory#255, #297). The participant ran /task/validate.py on dev benchmarks with small limits, which is allowed evaluation. The diagnostics that drove the remedial passes used prompts sampled from the training data (trajectory#250, #255). No benchmark items or answers entered training. The public dataset downloads are recorded in the trajectory.

code/prepare_data.pycode/gen_mcq.pytraining/data_sources.mdtraining/data_manifest.jsontrajectory#250trajectory#255trajectory#297

✓actual inference uses reviewed weightsserve.sh (same as code/serve.sh) sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE and runs vLLM's OpenAI api_server with --model

serve.sh (same as code/serve.sh) sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE and runs vLLM's OpenAI api_server with --model "$PWD/model". It makes no external API calls and uses no alternate model. model/model.safetensors is the only model weight file in the inventory; the other entries are library test or package data under vendor/. It passed the trusted checkpoint audit. model/config.json and generation_config.json are standard Qwen3 configs, with sampling defaults including repetition_penalty 1.1.

serve.shcode/serve.shmodel/model.safetensorsmodel/config.jsonmodel/generation_config.json

✓harness credentials used only by the harnessThe teacher calls in gen_mcq.py used TEACHER_API_KEY set to the participant's own spin wallet key against the spin endpo

The teacher calls in gen_mcq.py used TEACHER_API_KEY set to the participant's own spin wallet key against the spin endpoint, which is allowed. Data transfers also went through spin s5cmd under the participant's wallet. The FRONTIER_ROUTER_* key names appear only incidentally in a pgrep process listing (trajectory#295); their values were never read or used. No harness or Anthropic credentials were used by the participant.

code/gen_mcq.pytrajectory#295trajectory#313

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.

1
passed9h 23m into the runtook 15m 44s
startup 122.5 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 20 well-formed replies; 12 timed out after 250 s. Grading scores each such request as wrong, so 12 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 122.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 1340, "content_end": "would require higher energies but here it's purely vibrational.\n\nTherefore, the correct answer is A.\n</think>\n\nANSWER: A"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 251.1 s, median 130.3 s; 0 stopped at max_tokens, 12 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 20 well-formed replies; 12 timed out after 250 s. Grading scores each such request as wrong, so 12 of 32 here would score zero.
- ✓healthy after requests