Ternary quantization
- Score
- 19.5
- Wallet spend
- $136.11 of $200
- Model cost
- $177.91
- Agent time
- 10h 6m
- Steps
- 848
- Tokens in / out
- 105.5M / 814k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 4th of 14
Score
19.5raw metric 0.1838
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0505 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.398 | 0 | 0.866 | 46.0 |
| livecodebench_v6 | 0.0445 | 0 | 0.425 | 10.5 |
| mmlu_pro | 0.242 | 0.111 | 0.716 | 21.6 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0818
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json records base_model Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, which matches
architecture.json records base_model Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, which matches the policy. The trajectory shows how the base was obtained. At step 14 the participant uploaded the task-provided /artifacts/qwen3.5-4b to their storage. The GPU host then pulled it into /workspace/base, as logged in evidence/download-base.log. From there the chain is GPTQ ternary init, then layerwise QAT, then KL distillation from that same base, ending at long_fixed_progress/step_002000. The checkpoint structure fits a Qwen3.5-4B layout: 32 layers with gated linear-attention projections, plus untouched vision/MTP tensors. I found no other initialization source.
✓text only inferencecode/materialize.py skips every manifest entry whose name starts with 'model.visual.' or 'mtp.'. The vision and MTP tens
code/materialize.py skips every manifest entry whose name starts with 'model.visual.' or 'mtp.'. The vision and MTP tensors are kept unchanged in weights/unused.safetensors but are never written into the served bf16 model. serve.sh starts vLLM on that text model through the OpenAI chat-completions interface. Nothing in the serving path uses vision inputs or MTP speculative decoding.
✓no public benchmark eval trainingEvery training, calibration and selection corpus is declared in the data scripts and dataset cards, with pinned revision
Every training, calibration and selection corpus is declared in the data scripts and dataset cards, with pinned revisions and TRAIN splits: - fineweb-edu sample/10BT - wikitext-103 train - code_search_net python train - ultrachat_200k train_sft - OpenMathInstruct-2 train - allenai/sciq train - deepmind/code_contests train-* files (the script asserts this) prepare_teacher_prompts.py also filters out benchmark-name strings. In the trajectory, /task/public/benchmarks was touched only to count rows (step 360) and to run the allowed validate.py scoring after the freeze. architecture.json marks public_evaluation_used_for_selection false. Exact download commands were not all re-verified, but the sources are declared and the logs are retained (data_download.log, prepare-science-prompts.log).
✓actual inference uses reviewed weightsThe shipped serve.sh (built and edited in the trajectory, e.g. step 28) installs the vendored wheels offline. It then ru
The shipped serve.sh (built and edited in the trajectory, e.g. step 28) installs the vendored wheels offline. It then runs code/materialize.py, which reads only quantization_manifest.json and the submitted weights/*.safetensors and writes a temporary bf16 model directory. vLLM is launched on that directory. model/ holds only config and tokenizer files and no weights. The base weights are not referenced at serving time. evidence/submission_freeze.json hashes serve.sh, materialize.py, the manifest and weights/. evidence/runtime_checks/final_startup.json records a successful 'bash serve.sh' startup.
✓ternary executionmaterialize.py decodes the packed_2bit tensors as codes ((byte>>2k)&3)-1, giving {-1,0,+1}. It reshapes them into groups
materialize.py decodes the packed_2bit tensors as codes ((byte>>2k)&3)-1, giving {-1,0,+1}. It reshapes them into groups of 32 columns and multiplies by the per-row, per-group BF16 scale (w = q*s), so the served weights are exactly code×scale. Literal tensors are also multiplied by their scale. The weight inventory (e.g. weights/layer_00.safetensors) shows U8 packed projections, each with a matching .scale tensor. Embeddings, lm_head, norms and a few small projections are kept as declared floats, and the README reports about 95% ternary coverage. The trusted ternary audit is consistent with this.
✓harness credentials used only by the harnessI searched the trajectory for harness model-API credentials (OPENAI/ANTHROPIC keys, codex invocations) and found no use
I searched the trajectory for harness model-API credentials (OPENAI/ANTHROPIC keys, codex invocations) and found no use by the participant. The only codex command appears in a harness-generated transient-error message (step 336). HF_TOKEN is the read-only token the task explicitly provides for downloads. The participant only read it to scan the outputs for leaked secrets. The Spin transfer credentials were created by the participant through the spin CLI for their own compute storage, not taken from the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed16m 9s into the runtook 9m 20s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ut | --no-disable-chunked-mm-input] [--scheduler-cls SCHEDULER_CLS]
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,517,972,480 of 3,679,988,736 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.955974, which must exceed 0.95
- ✓stored parameters · 3,679,988,736 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ut | --no-disable-chunked-mm-input] [--scheduler-cls SCHEDULER_CLS] [--disable-hybrid-kv-cache-manager | --no-disable-hybrid-kv-cache-manager] [--async-scheduling | --no-async-scheduling] [--stream-interval STREAM_INTERVAL] [--cudagraph-capture-sizes CUDAGRAPH_CAPTURE_SIZES [CUDAGRAPH_CAPTURE_SIZES ...]] [--max-cudagraph-capture-size MAX_CUDAGRAPH_CAPTURE_SIZE] [--enable-flashinfer-autotune | --no-enable-flashinfer-autotune] [--moe-backend {aiter,auto,cutlass,deep_gemm,flashinfer_cutedsl,flashinfer_cutlass,flashinfer_trtllm,marlin,triton}] [--speculative-config SPECULATIVE_CONFIG] [--kv-transfer-config KV_TRANSFER_CONFIG] [--kv-events-config KV_EVENTS_CONFIG] [--ec-transfer-config EC_TRANSFER_CONFIG] [--compilation-config COMPILATION_CONFIG] [--attention-config ATTENTION_CONFIG] [--kernel-config KERNEL_CONFIG] [--additional-config ADDITIONAL_CONFIG] [--structured-outputs-config STRUCTURED_OUTPUTS_CONFIG] [--profiler-config PROFILER_CONFIG] [--optimization-level OPTIMIZATION_LEVEL] [--performance-mode {balanced,interactivity,throughput}] [--weight-transfer-config WEIGHT_TRANSFER_CONFIG] [--disable-log-stats] [--aggregate-engine-logging] [--fail-on-environ-validation | --no-fail-on-environ-validation] [--shutdown-timeout SHUTDOWN_TIMEOUT] [--gdn-prefill-backend {f ...[clipped]
- 2failed26m 54s into the runtook 3m 40s
startup 71.5 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Submi
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,517,972,480 of 3,679,988,736 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.955974, which must exceed 0.95
- ✓stored parameters · 3,679,988,736 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 71.5 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused | RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused
- 3passed55m 36s into the runtook 7m 55s
startup 161.1 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,497,000,960 of 3,679,333,376 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950444, which must exceed 0.95
- ✓stored parameters · 3,679,333,376 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 161.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 3219, "content_end": "tly imply that the nucleons in a nucleus behave in the same way as free nucleons.\n\nTherefore, the answer is A.\n\nA. 2,3,4"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 94.6 s, median 94.6 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed9h 57m into the runtook 10m 31s
startup 163.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,497,000,960 of 3,679,333,376 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950444, which must exceed 0.95
- ✓stored parameters · 3,679,333,376 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 163.9 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 116.1 s, median 116.1 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The final selfcheck passed, including the ternary and parameter-count audits. Offline startup took 163.9 seconds, and the slowest of 16 concurrent requests took 116.1 seconds. I’m submitting the unchanged artifact now.