Ternary quantization
- Score
- 5.4
- Wallet spend
- $49.45 of $50
- Model cost
- $12.59
- Agent time
- 5h 53m
- Steps
- 333
- Tokens in / out
- 33.3M / 416k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 5th of 14
Score
5.4raw metric 0.08268
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0404 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.182 | 0 | 0.866 | 21.0 |
| livecodebench_v6 | 0.00190 | 0 | 0.425 | 0.4 |
| mmlu_pro | 0.106 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0071
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startTraining starts from the pinned base model. The rental bootstrap runs snapshot_download('Qwen/Qwen3.5-4B', revision='851
Training starts from the pinned base model. The rental bootstrap runs snapshot_download('Qwen/Qwen3.5-4B', revision='851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a') into /root/model (trajectory#70; the export bootstrap repeats it). In code/qat_train.py, build_text_model loads those original shards through model.safetensors.index.json for both the trainable student latents and the frozen bf16 teacher. Export used export_weights.py --model /root/model together with the trained latent. architecture.json declares the same repo and revision, and the task's /artifacts copy carries that revision (trajectory#1). No other initialization was found.
✓text only inferenceserve.sh execs code/server.py, which calls materialize.load_ternary_model. That function builds Qwen3_5ForCausalLM from
serve.sh execs code/server.py, which calls materialize.load_ternary_model. That function builds Qwen3_5ForCausalLM from the text config only and loads only model.language_model.* manifest entries. server.py flattens list message content to text parts, so no image path exists. The vision and MTP weights ship in bf16 but the serving path never loads them.
✓no public benchmark eval trainingThe training corpora, per evidence/data_manifest.json and code/prep_data*.py, are all public training splits or general
The training corpora, per evidence/data_manifest.json and code/prep_data*.py, are all public training splits or general corpora: fineweb-edu sample-10BT, wikipedia 20231101.en, codeparrot github-code (partial train parquet), and the train splits of MetaMathQA, MathInstruct and NuminaMath-TIR. The abandoned prep_data2 attempt is also recorded. The bootstrap pulls training data into /root/data/packed, while benchmark files go into a separate bench directory (trajectory#70). The GPQA/LCB files under /task/public were used only for validate.py runs and serving and debug generations. The participant did generate on two GPQA items while choosing the decoding knobs (trajectory#249, trajectory#265, trajectory#266, trajectory#267). That choice rested on output-format behaviour (EOS and answer-marker emission, plus a non-benchmark Fibonacci code prompt), not on benchmark accuracy (both answers were wrong), and no weights were trained or calibrated on benchmark data. This is noted but is not an established training or selection violation.
✓actual inference uses reviewed weightsCall graph: serve.sh → code/server.py → materialize.load_ternary_model(args.dir, config_dir). That function reads quanti
Call graph: serve.sh → code/server.py → materialize.load_ternary_model(args.dir, config_dir). That function reads quantization_manifest.json and the shipped weights/text_packed-00001.safetensors, weights/text_scales-00001.safetensors, weights/text_float.safetensors and weights/embed_head.safetensors, which are the same files the trusted ternary audit read (passed, 0 unreferenced parameters). code/fast_engine.py only wraps that same model object in CUDA graphs; it loads no other weights and no original checkpoint.
✓ternary executionPacked 2-bit codes are unpacked as codes-1 ∈ {-1,0,+1} times per-group (32) fp32 scales in materialize.unpack_dequant_gp
Packed 2-bit codes are unpacked as codes-1 ∈ {-1,0,+1} times per-group (32) fp32 scales in materialize.unpack_dequant_gpu, and those values are the linear weights the model executes with. The trusted audit reports ternary fraction 0.9684 with the parameter cap passed. Embedding, head (tied), norms and other small tensors stay floating point, as permitted.
✓harness credentials used only by the harnessSearches of the trajectory found no use of harness or judge API keys, no direct model-API calls, and no use of the claud
Searches of the trajectory found no use of harness or judge API keys, no direct model-API calls, and no use of the claude, codex or similar CLIs. Credential use was limited to the Spin CLI for compute and storage (spin session S3 credentials for the participant's own data bucket, trajectory#70) and HF_TOKEN for downloads from Hugging Face, both of which are permitted.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 38m into the runtook 4m 34s
startup 157.3 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error": "\'Request\' object has no attribute \'result\'"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "\'Request\' object has no attribute \'result\'"}' | Run
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 157.3 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error": "\'Request\' object has no attribute \'result\'"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "\'Request\' object has no attribute \'result\'"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "\'Request\' object has no attribute \'result\'"}'
- 2passed2h 50m into the runtook 14m 23s
startup 152.7 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 152.7 s
- ✓GPQA request as graded · {"note": "no reply within 300 s: grading scores a timed-out request as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission returned HTTP 500: b'{\"error\": \"generation timeout\"}'"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 296.2 s, median 295.2 s; 0 stopped at max_tokens, 16 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
- 3passed5h 47m into the runtook 6m 55s
startup 145.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 145.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 106.7 s, median 53.7 s; 7 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Budget exhausted as planned ($0.56 left, all GPU work complete). Submitting: