Ternary quantization
- Score
- 29.3
- Wallet spend
- $31.41 of $50
- Model cost
- $97.28
- Agent time
- 5h 46m
- Steps
- 200
- Tokens in / out
- 51.2M / 255k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 2nd of 14
Score
29.3raw metric 0.2796
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.217 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.456 | 0 | 0.866 | 52.7 |
| livecodebench_v6 | 0.134 | 0 | 0.425 | 31.5 |
| mmlu_pro | 0.311 | 0.111 | 0.716 | 33.1 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2287
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares the base model as Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In tr
architecture.json declares the base model as Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In trajectory#26 the participant downloads exactly that revision with `hf download Qwen/Qwen3.5-4B --revision 851bf6e8... --local-dir /root/work/qwen3.5-4b`. quant/model_utils.py loads the model.language_model.* weights from that local checkpoint. quant/quantize.py runs GPTQ ternary initialization on those original fp weights, followed by Block-AP and QAT. The shipped config is Qwen3_5ForConditionalGeneration and keeps the same tensor structure. No other initialization source was found.
✓text only inferenceserve.sh starts vLLM with --language-model-only. serve/proxy.py only renders the chat template and forwards text to the
serve.sh starts vLLM with --language-model-only. serve/proxy.py only renders the chat template and forwards text to the local /v1/completions backend. Vision and MTP tensors exist only as unchanged excluded parameters and are not used for serving.
✓no public benchmark eval trainingquant/calib_data.py and evidence/data_manifest.json declare only train splits: gsm8k train, hendrycks_math train, ARC-Ch
quant/calib_data.py and evidence/data_manifest.json declare only train splits: gsm8k train, hendrycks_math train, ARC-Challenge train, MMLU auxiliary_train, sciq train, code_contests train, mbpp train, ultrachat_200k train_sft and fineweb-edu sample-10BT. The download commands are the load_dataset calls in the script; the logs were not independently verified. Distillation responses came from the bf16 base model on a local vLLM. In the trajectory, /task/public/benchmarks/gpqa_diamond.csv is used only by eval_gpqa.py to score local servers (trajectory#35-51, trajectory#91-139). It was not used for calibration or training.
✓actual inference uses reviewed weightsserve.sh runs serve/dequantize.py, which reads quantization_manifest.json and the audited weights/*.safetensors (lm_00..
serve.sh runs serve/dequantize.py, which reads quantization_manifest.json and the audited weights/*.safetensors (lm_00..lm_28, embed and extra) and writes /tmp/model. vLLM is then served with --model /tmp/model. The trusted ternary audit covers these same files, with no unreferenced or omitted tensors. proxy.py calls only the local vLLM backend.
✓ternary executionThe trusted audit passed, with a ternary fraction of 0.968 and storage within the limit. Packed 2-bit codes with BF16 gr
The trusted audit passed, with a ternary fraction of 0.968 and storage within the limit. Packed 2-bit codes with BF16 group-32 scales are stored for the decoder linears. dequantize.py rebuilds each weight as scale×code in {-1,0,+1}, which is exact in bf16, so the served linear weights are ternary-valued. The float exceptions (embedding, lm_head, norms, conv1d, in_proj_a/b) are declared in architecture.json and accounted for in the audit.
✓harness credentials used only by the harnessSearches of the trajectory for API keys, base URLs, external LLM calls and credential use found only the harness's own t
Searches of the trajectory for API keys, base URLs, external LLM calls and credential use found only the harness's own task and spin documentation. All model calls by the participant went to local vLLM servers, and no misuse of harness credentials was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed4h 22m into the runtook 4m 20s
startup 82.6 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"Server error \'500 Internal Server Error\' for url \'http://127.0.0.1:8001/v1/completions\'\\nFor more information check: https://developer.mozilla | Runti
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 82.6 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"Server error \'500 Internal Server Error\' for url \'http://127.0.0.1:8001/v1/completions\'\\nFor more information check: https://developer.mozilla | RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"All connection attempts failed","type":"server_error"}}' | RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"All connection attempts failed","type":"server_error"}}'
- 2passed4h 32m into the runtook 6m
startup 169.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 169.6 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 25.1 s, median 24.1 s; 6 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed5h 47m into the runtook 4m 44s
startup 84.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 84.5 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 11947, "content_end": "ke in my manual calculation or the pattern is simpler. Let's look at the numbers again.\n115 = 5 * 23\n176 = 12 * 14.66..."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.2 s, median 22.3 s; 5 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Final selfcheck passed on the submitted folder (weight audit, startup 84.5 s, all request gates), the GPU lease is stopped, and the submission is complete. Submitting now.