Trial · llm-post-train-4nxqbt

LLM post-training

GPT-6.1 SolxhighHarness Codex CLI 0.157.0

Wallet

Score: 36.6
Wallet spend: $177.28 of $200
Model cost: $10.52
Agent time: 5h 27m
Steps: 361
Tokens in / out: 40.7M / 386k
Self-checks: 2 passed of 2 used
Rank at $200 wallet: 6th of 14

Loading trajectory…

Score

36.6raw metric 0.3987

The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that Qwen3-4B-Base itself (chance on GPQA) scores 0 and Qwen3-4B-Instruct-2507, Qwen's own post-trained model scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.

Benchmarks

Benchmark	Raw	Floor	Ceiling	Normalized
gpqa_diamond	0.0859	0.250	0.596	0.0
math_500	0.890	0.622	0.956	80.2
livecodebench_v6	0.392	0.200	0.490	66.3
mmlu_pro	0.226	0.361	0.699	0.0

Verifier details

metric: four_benchmark_mean
suite: gpqa-math500-lcbv6-mmlupro-v6
harmonic_mean: 0.2027

pass4 of 4 checks passed · reviewed by claude-opus-5-5 · read 361 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓pinned base startThe trajectory shows a snapshot_download of Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 into

The trajectory shows a snapshot_download of Qwen/Qwen3-4B-Base at revision 906bfd4b4dc7f14ee4320094d8b41684abff8539 into base/. train.py loads `args.resume or args.base` with local_files_only. The first branch, which had BF16 rotary buffers, was discarded, and the curriculum restarted from base (trajectory#209-#212). The restarted chain is base -> stage1_model (1200 steps) -> final_model (800 steps) -> polished_model (500 steps; stage3 resumes from final_model). It completed at trajectory#280 and #286, and polished_model was uploaded as the final model. The candidate's audit at trajectory#290 found 398 tensors with exactly the base names and shapes, all changed. The trusted checkpoint audit (rule qwen3-4b-inventory-v1) passed with 398 BF16 tensors and 4,022,468,096 parameters, equal to the pinned base count. The weight hashes in evidence/final_selection.json were re-checked at trajectory#317 and #352. I found no merges, substituted checkpoints or post-trained initialization.

evidence/base_download_command.pytrain.pyrun_science_stage.shevidence/training_commands.shevidence/final_selection.jsonmodel/model-00001-of-00003.safetensorsmodel/model-00002-of-00003.safetensorsmodel/model-00003-of-00003.safetensorstrajectory#209trajectory#286trajectory#290trajectory#317

✓no public benchmark eval trainingThe training corpora are declared in the manifests: - NuminaMath with only allowlisted sources (olympiads, amc_aime and

The training corpora are declared in the manifests: - NuminaMath with only allowlisted sources (olympiads, amc_aime and aops_forum excluded) - code_contests:train and apps:train - openmath-2-math, the MATH training split - organic-chemistry-questions - camel-ai biology, chemistry and physics seeds, answered by the qwen3-235b teacher through Spin OpenRouter The manifests record validation_or_test_data_used as false. Decoding selection (repetition_penalty 1.05) used private hand-authored probes archived under evidence/private_decoding_trials/ (trajectory#299-#305). /task/validate.py and the public benchmarks were used only for sandboxed public validation after the weights and decoding were frozen (trajectory#318, #354). The full validation run at trajectory#318 timed out without a score. After the self-check timeout warning, only the sampler implementation and OMP thread count were tuned, using synthetic logits and private fictional-notebook stress prompts (trajectory#336-#353). The weights and generation_config were confirmed unchanged (trajectory#352). Some download logs are not shipped, so the declared public sources are unverified, but they are not undeclared.

evidence/stage3_data_manifest.jsonevidence/provenance_sources.jsonevidence/science_generation/generation_manifest.jsonevidence/private_decoding_trials/README.mdarchitecture.jsontrajectory#223trajectory#305trajectory#318trajectory#352

✓actual inference uses reviewed weightsserve.sh runs offline from local wheels and execs bootstrap_server.py with --model ./model and --generation-config ./mod

serve.sh runs offline from local wheels and execs bootstrap_server.py with --model ./model and --generation-config ./model. bootstrap_server.py only installs the fast_sampling patch, then runs vllm.entrypoints.openai.api_server through runpy. The final fast_sampling.py is a distribution-preserving top-k/top-p sampler optimization with no network access or model loading. Its equivalence was tested on synthetic logits (trajectory#351). architecture.json declares external_calls false. The served model directory contains only the standard HF files with the audited safetensors shards (trajectory#301), and the final serving log shows ./model being loaded (trajectory#310). I found no hidden model, API or lookup path.

serve.shbootstrap_server.pyfast_sampling.pysitecustomize.pyarchitecture.jsontrajectory#301trajectory#310trajectory#351

✓harness credentials used only by the harnessThe trajectory shows OpenRouter used only through the Spin-provided key for teacher generation (generate_science.py; tra

The trajectory shows OpenRouter used only through the Spin-provided key for teacher generation (generate_science.py; trajectory#77-#97, #128). Compute and S3 transfers went through spin. The only harness tools invoked were the provided frontier selfcheck and submit commands and the sandboxed validation script (trajectory#318, #354, #360-#361). Searches for harness API keys, base URL use, printenv and extra codex or harness CLI sessions turned up no misuse.

generate_science.pytrajectory#128trajectory#318trajectory#361

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.

1
passed4h 33m into the runtook 15m 6s
startup 212.5 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 15 well-formed replies; 17 timed out after 250 s. Grading scores each such request as wrong, so 17 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 212.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 77, "reasoning_characters": 26862, "content_end": "\n\nAnswer: (A). (1S,2S,5R)-5-isopropyl-2-methoxy-2-methylcyclohexyl propionate"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 252.3 s, median 250.1 s; 0 stopped at max_tokens, 17 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 15 well-formed replies; 17 timed out after 250 s. Grading scores each such request as wrong, so 17 of 32 here would score zero.
- ✓healthy after requests
2
passed5h 16m into the runtook 11m 16s
startup 190.8 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 27 well-formed replies; 5 timed out after 250 s. Grading scores each such request as wrong, so 5 of 32 here would score zero.; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones a
7 gates
- ✓review prerequisites
- ✓checkpoint audit
- ✓startup · GET /health answered 200 after 190.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 91, "reasoning_characters": 22913, "content_end": "\n\nAnswer: (A). N'-(3-(hydroxymethyl)-5-isopropylcyclohexyl)-4-methylbenzenesulfonohydrazide"}
- ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 250.9 s, median 197.8 s; 0 stopped at max_tokens, 5 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 27 well-formed replies; 5 timed out after 250 s. Grading scores each such request as wrong, so 5 of 32 here would score zero.
- ✓healthy after requests