Ternary quantization
- Score
- 16.7
- Wallet spend
- $13.96 of $50
- Model cost
- $17.34
- Agent time
- 3h 14m
- Steps
- 354
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 4th of 19
Score
16.7raw metric 0.2058
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.242 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.276 | 0 | 0.866 | 31.9 |
| livecodebench_v6 | 0.0427 | 0 | 0.425 | 10.0 |
| mmlu_pro | 0.262 | 0.111 | 0.716 | 24.9 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.1143
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startAt trajectory step 44 the participant ran snapshot_download for Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e
At trajectory step 44 the participant ran snapshot_download for Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /home/dev/qwen3.5-4b. src/train_and_export.py loads the base model from that local snapshot and sets BASE_MODEL and BASE_REVISION to the pinned values. It then GPTQ-ternarizes, distills and exports those weights. architecture.json and the exported training summary record the same base model and revision. The quantization manifest lists every language-model tensor of the Qwen3.5 architecture. I found no other starting checkpoint.
✓text only inferencesrc/server.py builds AsyncEngineArgs with language_model_only=True and limit_mm_per_prompt set to 0 for image and video.
src/server.py builds AsyncEngineArgs with language_model_only=True and limit_mm_per_prompt set to 0 for image and video. src/ternary_loader.py skips the model.visual.* and mtp.* tensors. serve.sh only runs src/server.py over the weights directory. Requests are handled as text chat and completion only.
✓no public benchmark eval trainingdata_manifest.json, architecture.json and src/generate_teacher_data.py declare that only train splits were used: ARC-Cha
data_manifest.json, architecture.json and src/generate_teacher_data.py declare that only train splits were used: ARC-Challenge train, NuminaMath-CoT train, Magicoder train and wikitext-2-raw-v1 train. The download recipes are retained. Teacher completions come from the pinned base model. Checkpoint selection uses held-out slices of those same train corpora. The wikitext test and validation splits were produced automatically by the dataset download, but the trajectory shows they were not used. I found no use of benchmark eval or test data or of task dev data.
✓actual inference uses reviewed weightsserve.sh runs src/server.py with --model-path pointing at the weights directory. The server sets load_format='ternary_ma
serve.sh runs src/server.py with --model-path pointing at the weights directory. The server sets load_format='ternary_manifest'. That loader is registered by src/ternary_loader.py, which is also imported via sitecustomize.py. The loader reads quantization_manifest.json and yields tensors only from weights/model-00001-of-00002.safetensors and weights/model-00002-of-00002.safetensors. The trusted ternary audit covers these same files, with no unreferenced or omitted tensors. There are no other weight files and no uninspected files.
✓ternary executionIn quantization_manifest.json the transformer linear weights are packed_2bit entries with per-group bf16 scales. src/ter
In quantization_manifest.json the transformer linear weights are packed_2bit entries with per-group bf16 scales. src/ternary_loader.py decodes them as ((packed>>shift)&3)-1, giving values in {-1,0,1}, and multiplies by the group scale, so the served weights are exactly ternary times scale. The trusted audit passed with a ternary fraction of 0.954 and lists only embeddings, lm_head, norms and similar tensors as float exclusions.
✓harness credentials used only by the harnessThe trajectory shows the antigravity/gemini harness making its own model calls. Searches for API keys, base URLs and Gem
The trajectory shows the antigravity/gemini harness making its own model calls. Searches for API keys, base URLs and Gemini/OpenRouter credentials found only spin CLI help text. The spin compute and s5cmd transfer sessions used in the trajectory are the participant's own compute and transfer tools, not harness model credentials. Neither the shipped code nor serve.sh calls an external LLM API.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 31m into the runtook 5m 26s
startup 120.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,470,786,560 of 3,627,396,096 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.956826, which must exceed 0.95
- ✓stored parameters · 3,627,396,096 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 120.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 13535, "content_end": "d charge and the 13th charge is 2m.\n * This implies the 13th charge is at the center of the\n\nANSWER: 2.822 x 10^-26"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.8 s, median 26.8 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed3h 9m into the runtook 4m 53s
startup 102.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,460,300,800 of 3,627,068,416 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.954021, which must exceed 0.95
- ✓stored parameters · 3,627,068,416 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 102.7 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 15918, "content_end": "Let's assume the starting compound is a specific compound.\n * Let's check the structure of the compound.\n\nANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 26.8 s, median 25.6 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests