Ternary quantization
- Score
- 28.9
- Wallet spend
- $73.29 of $200
- Model cost
- $263.42
- Agent time
- 9h 15m
- Steps
- 206
- Tokens in / out
- 58.6M / 325k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 3rd of 14
Score
28.9raw metric 0.2849
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.303 | 0.250 | 0.571 | 16.5 |
| math_500 | 0.468 | 0 | 0.866 | 54.0 |
| livecodebench_v6 | 0.0351 | 0 | 0.425 | 8.3 |
| mmlu_pro | 0.334 | 0.111 | 0.716 | 36.7 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.1083
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startIn trajectory step 23, the participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13c
In trajectory step 23, the participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /workspace/model. The download output appears in steps 33-50. The training code loads only /workspace/model: model_utils.load_text_model calls Qwen3_5ForCausalLM.from_pretrained, and that is used by sens.py and the KD/GPTQ path. The final phase-4 run (phase4.sh) starts from earlier exports of that same lineage. export.py pins BASE_REV and copies the non-ternary tensors from the original model. architecture.json declares the same base model and revision.
✓text only inferenceserve.sh starts vLLM with --language-model-only on the dequantized checkpoint. code/proxy.py applies the chat template t
serve.sh starts vLLM with --language-model-only on the dequantized checkpoint. code/proxy.py applies the chat template to text messages only and forwards only to the local vLLM /v1/completions endpoint. The vision and MTP weights are unused at serving time and are excluded from the audit.
✓no public benchmark eval trainingtraining/DATA_MANIFEST.md declares every training and calibration source: - fineweb-edu sample-10BT train - OpenThoughts
training/DATA_MANIFEST.md declares every training and calibration source: - fineweb-edu sample-10BT train - OpenThoughts3 train - teacher generations from the train splits of OpenR1-Math, OpenCodeReasoning split_0, tulu-3-sft-mixture, MMLU auxiliary_train, ARC-Challenge, OpenBookQA and SciQ generate.py, prep_data.py and phase4.sh load exactly these sources. The public GPQA-Diamond and LCB files appear in the trajectory only in evaluation packaging and validate.py runs, never in training. The downloads went through load_dataset; no retained per-dataset download logs were checked, so those are recorded as unverified but declared public sources.
✓actual inference uses reviewed weightsserve.sh runs code/dequant.py on quantization_manifest.json. dequant.py reads only the tensors referenced by the manifes
serve.sh runs code/dequant.py on quantization_manifest.json. dequant.py reads only the tensors referenced by the manifest from weights/model-00001-of-00002.safetensors and weights/model-00002-of-00002.safetensors, writes /tmp/model_bf16, and vLLM serves that directory with HF_HUB_OFFLINE=1. No alternate model or remote endpoint is used. These are the same files the trusted ternary audit inspected (audited_storage, 0 unreferenced parameters).
✓ternary executionThe trusted audit passed with a ternary fraction of 0.963 and the parameter cap passed. dequant.py rebuilds each packed_
The trusted audit passed with a ternary fraction of 0.963 and the parameter cap passed. dequant.py rebuilds each packed_2bit/literal linear exactly as W = scale[o, i//32] * t[o,i], with t in {-1,0,+1}. Only the excluded tensors (embedding, visual, MTP) and the two declared layer-0 float linears stay in bf16, so serving executes the audited ternary weights.
✓harness credentials used only by the harnessThe trajectory shows HF_TOKEN used only to check it was present (step 21) and for downloads, passed to the Spin GPU box
The trajectory shows HF_TOKEN used only to check it was present (step 21) and for downloads, passed to the Spin GPU box for hf download and load_dataset (step 23). Spin was used for compute and storage. Searches for harness model-API keys, base URLs and harness CLIs found no use by the participant. All model calls in the shipped code, including teacher generation, go to a local vLLM server.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 40m into the runtook 7m 19s
failed startup: Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status":"starting"}'); end of its log: _llm = AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 248, i
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status":"starting"}'); end of its log: _llm = AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 248, in from_vllm_config return cls( ^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 180, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 167, in make_async_mp_client return AsyncMPClient( ^^^^^^^^^^^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 1100, in __init__ super().__init__( File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 674, in __init__ with launch_core_engines( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1268, in launch_core_engines wait_for_engine_startup( File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1348, in wait_for_engine_startup raise RuntimeError( RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1268, in launch_core_engines wait_for_engine_startup( File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1348, in wait_for_engine_s ...[clipped]
- 2failed2h 49m into the runtook 7m 2s
failed startup: Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status":"starting"}'); end of its log: kv_cache_config = self._initialize_kv_caches(vllm_config) 80:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] ^^^^^^^^^^^^^^^^^^^^^^
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status":"starting"}'); end of its log: kv_cache_config = self._initialize_kv_caches(vllm_config) 80:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 82:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] return func(*args, **kwargs) 83:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] ^^^^^^^^^^^^^^^^^^^^^ 84:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 357, in _initialize_kv_caches 85:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] self.model_executor.compile_or_warm_up_model() 86:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 128, in compile_or_warm_up_model 87:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] compilation_times: list[CompilationTimes] = self.collective_rpc( 88:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] ^^^^^^^^^^^^^^^^^^^^ 89:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 103, in collective_rpc 90:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] result = run_method(self.driver_worker, method, args, kwargs) 91:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 92:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 508, in run_method 93:(EngineCore pid=573) ERROR 10-06 01:14:38 [core.py:1433] return func(*args, **kwargs) File "/tmp/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1348, in wait_for_engine_s ...[clipped]
- 3passed3h 13m into the runtook 6m 34s
startup 228.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 228.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 1410, "reasoning_characters": 4511, "content_end": "sted format:\nThe user is asking which of the symmetries must be respected by all ...[clipped]
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 24.6 s, median 23.7 s; 12 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed9h 10m into the runtook 6m 20s
startup 203.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,544,186,880 of 3,680,807,936 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.962883, which must exceed 0.95
- ✓stored parameters · 3,680,807,936 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 203.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 2751, "reasoning_characters": 10944, "content_end": "reaction of a **secondary alcohol** with NaNO2 and HCl is a **reduction\n\nFinal answer in the requested format:\nANSWER: D"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 24.1 s, median 23.5 s; 1 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 07:39:58 UTC. The session is over and /workspace/output is being graded as submitted.
Final state of the submission:
- Weights: Qwen3.5-4B text model with all 200 linear projections ternary (2-bit packed codes, bf16 scale per 32 input weights), layer-0
in_proj_z/out_projkept in bf16; ternary fraction 96.3% of audited parameters, stored parameters 1.032× the original. Weight audit passed on the grading image. - Serving:
serve.sh(offline uv install of vendored wheels, ternary-to-bf16 expansion, vLLM 0.31 with Triton kernels and prewarmed caches, thinking-budget front end). Final selfcheck: startup 204 s of 300, 16 concurrent graded-style requests answered in well-formed JSON. - Quality: the student retains most ability on easier training-split checks (~71-78% on GSM8K-train / MMLU-aux) but remains near chance on GPQA-Diamond and close to zero on LiveCodeBench; sub-2-bit QAT of a reasoning model needs far more training tokens than this budget allowed. All code, logs, data provenance and the quantization conventions are documented under
training/andREADME.md. - Resources: both GPU leases released; $73 of the $200 compute budget used.