Ternary quantization
- Score
- 4.6
- Wallet spend
- $166.71 of $200
- Model cost
- $11.52
- Agent time
- 10h 33m
- Steps
- 316
- Tokens in / out
- 37.8M / 330k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 7th of 14
Score
4.6raw metric 0.03223
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.00505 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0280 | 0 | 0.866 | 3.2 |
| livecodebench_v6 | 0.0645 | 0 | 0.425 | 15.2 |
| mmlu_pro | 0.0314 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0142
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen starttraining/prepare_data.py calls snapshot_download for Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0
training/prepare_data.py calls snapshot_download for Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. training/common.py load_base loads the model.language_model.* tensors from that local base. training/quantize.py builds the GPTQ ternary initialization from those weights, and training/distill.py starts the student from load_base plus apply_quantized. The base and revision in quantization_manifest.json and architecture.json match the pinned ones. The trajectory shows no other checkpoint source.
✓text only inferenceserve.sh runs vLLM with --language-model-only. materialize.py skips every tensor whose name starts with model.visual. or
serve.sh runs vLLM with --language-model-only. materialize.py skips every tensor whose name starts with model.visual. or mtp. The visual and MTP tensors exist only in weights/unused_original.safetensors, which nothing on the serving path ever loads.
✓no public benchmark eval trainingEvery training, calibration and head-fit corpus is a declared public training split: wikitext-103 train, Magicoder-OSS-I
Every training, calibration and head-fit corpus is a declared public training split: wikitext-103 train, Magicoder-OSS-Instruct decontaminated train, OpenMathInstruct-2 train_1M and ultrachat_200k train_sft. Each has a pinned revision and download command in evidence/data_manifest.json. Teacher generations use only prompts from those training splits, and they come from the local original Qwen teacher (evidence/generated/teacher_data_manifest.json, training/prepare_teacher.py, training/generate_teacher.py). The distillation length is a fixed 8500-step cutoff. The trajectory shows /task/validate.py on the public benchmarks being run only once, on the finished submission (trajectory#299 onward); it did not feed into training or selection. Searches found no gsm8k, test-split or validation-split loading.
✓actual inference uses reviewed weightsserve.sh runs materialize.py with --source set to the submission root, then serves only /tmp/ternary-model. materialize.
serve.sh runs materialize.py with --source set to the submission root, then serves only /tmp/ternary-model. materialize.py builds that directory only from the tensors listed in quantization_manifest.json, read out of the audited weights/*.safetensors files. It decodes the packed codes with their scales and copies only config and tokenizer files. sitecustomize.py swaps one kernel forward function and touches no weights. Nothing on the serving path references /artifacts or the original Qwen weights.
✓ternary executionThe trusted ternary audit passed: ternary fraction 0.95070 above the 0.95 threshold, 3,727,993,856 stored parameters aga
The trusted ternary audit passed: ternary fraction 0.95070 above the 0.95 threshold, 3,727,993,856 stored parameters against a limit of 3,927,057,305, and no unreferenced tensors. The language layers store U8 packed 2-bit codes in {-1,0,+1} with BF16 group scales. materialize.py decodes them as code×scale, the standard dequantized way to run ternary weights. Only the embedding, lm_head, norms and the small SSM parameters stay in floating point, as the task allows.
✓harness credentials used only by the harnessTrajectory searches for OPENAI, BASE_URL, SPIN_BROKER_TOKEN and openrouter key match only documentation text from spin -
Trajectory searches for OPENAI, BASE_URL, SPIN_BROKER_TOKEN and openrouter key match only documentation text from spin --help and spin docs. No step misuses a credential. The teacher endpoint is a local vLLM server at 127.0.0.1:8001 running the original Qwen (training/teacher_server.sh, training/generate_teacher.py). HF_TOKEN and Spin were used only for permitted downloads and compute.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed15m 58s into the runtook 8m 26sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/frontier-selfcheck-y0jks6gb/submission/quantization_manifest.json'
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: FileNotFoundError: [Errno 2] No such file or directory: '/tmp/frontier-selfcheck-y0jks6gb/submission/quantization_manifest.json'
- 2failed29m 45s into the runtook 7m 43s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [--enable-chunked-prefill | --no-enable-chunked-prefill]
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,544,186,880 of 3,727,993,856 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950695, which must exceed 0.95
- ✓stored parameters · 3,727,993,856 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [--enable-chunked-prefill | --no-enable-chunked-prefill] [--disable-chunked-mm-input | --no-disable-chunked-mm-input] [--scheduler-cls SCHEDULER_CLS] [--disable-hybrid-kv-cache-manager | --no-disable-hybrid-kv-cache-manager] [--async-scheduling | --no-async-scheduling] [--stream-interval STREAM_INTERVAL] [--cudagraph-capture-sizes CUDAGRAPH_CAPTURE_SIZES [CUDAGRAPH_CAPTURE_SIZES ...]] [--max-cudagraph-capture-size MAX_CUDAGRAPH_CAPTURE_SIZE] [--enable-flashinfer-autotune | --no-enable-flashinfer-autotune] [--moe-backend {aiter,auto,cutlass,deep_gemm,flashinfer_cutedsl,flashinfer_cutlass,flashinfer_trtllm,marlin,triton}] [--speculative-config SPECULATIVE_CONFIG] [--kv-transfer-config KV_TRANSFER_CONFIG] [--kv-events-config KV_EVENTS_CONFIG] [--ec-transfer-config EC_TRANSFER_CONFIG] [--compilation-config COMPILATION_CONFIG] [--attention-config ATTENTION_CONFIG] [--kernel-config KERNEL_CONFIG] [--additional-config ADDITIONAL_CONFIG] [--structured-outputs-config STRUCTURED_OUTPUTS_CONFIG] [--profiler-config PROFILER_CONFIG] [--optimization-level OPTIMIZATION_LEVEL] [--performance-mode {balanced,interactivity,throughput}] [--weight-transfer-config WEIGHT_TRANSFER_CONFIG] [--disable-log-stats] [--aggregate-engine-logging] [--fail-on-environ-validation | --no-fail-on-e ...[clipped]
- 3failed46m 35s into the runtook 7m 33s
startup 153.8 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Subm
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,544,186,880 of 3,727,993,856 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950695, which must exceed 0.95
- ✓stored parameters · 3,727,993,856 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 153.8 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused | RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused
- 4passed1h 25m into the runtook 7m 14s
startup 184.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,544,186,880 of 3,727,993,856 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950695, which must exceed 0.95
- ✓stored parameters · 3,727,993,856 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 184.4 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.8 s, median 27.7 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 5passed9h 54m into the runtook 7m 29s
startup 171.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,544,186,880 of 3,727,993,856 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950695, which must exceed 0.95
- ✓stored parameters · 3,727,993,856 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 171.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.3 s, median 26.9 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- 95.07% ternary, with 4.42% parameter overhead.
- Offline H100 selfcheck passed; startup took 171.2 seconds.
- Training provenance, quantization code, weights, and dependencies are included.
- Public accuracy: GPQA 0.51%, LiveCodeBench 6.45%—significant accuracy loss remains despite recovery training.