Ternary quantization
- Score
- 2.1
- Wallet spend
- $38.39 of $50
- Model cost
- $10.91
- Agent time
- 8h 48m
- Steps
- 391
- Tokens in / out
- 43.4M / 358k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
2.1raw metric 0.05651
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0606 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0740 | 0 | 0.866 | 8.5 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.0914 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. train.py loads model.la
architecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. train.py loads model.language_model.* tensors from the local base artifact /artifacts/qwen3.5-4b. ternary.py initializes the ternary codes and scales from those original weights with a Lloyd fit, followed by sequential reconstruction and distillation. evidence/starting_artifact.json and the session (trajectory#6) record the same pinned artifact as the starting point. The final audit reports 312 exempt tensors exactly unchanged from the original (trajectory#388). I found no other initialization source.
✓text only inferenceserve.sh launches the vLLM api_server with --language-model-only on the model materialized by materialize.py. sitecustom
serve.sh launches the vLLM api_server with --language-model-only on the model materialized by materialize.py. sitecustomize.py only switches ChunkGatedDeltaRule to its native forward. The vision-tower tensors are stored unchanged in weights/exempt.safetensors but are not used for inference in language-model-only mode, and every serving request in the session is text-only.
✓no public benchmark eval trainingbuild_data.py uses only the training splits of fineweb-edu, ultrachat_200k (train_sft), MATH-lighteval and python_code_i
build_data.py uses only the training splits of fineweb-edu, ultrachat_200k (train_sft), MATH-lighteval and python_code_instructions_18k_alpaca, each at a pinned revision. select_checkpoint.py chooses the checkpoint by NLL on an internal subset of those training splits. calibrate_decoder.py asserts that every record is from a train or train_sft split and selected repetition penalty 1.2 with a termination and duplicate-n-gram rule on 24 training documents (trajectory#362, #369, #372; evidence/decoder_selection.json). GPQA validate.py was run only as a diagnostic or final evaluation (trajectory#381, #386). Neither weights nor decoding parameters changed after that evaluation (trajectory#387-390). The data sources are declared; download logs are retained only for the decoder-calibration data, so the other provenance is unverified but not contradicted.
✓actual inference uses reviewed weightsserve.sh runs materialize.py, which builds every served tensor only from quantization_manifest.json entries pointing to
serve.sh runs materialize.py, which builds every served tensor only from quantization_manifest.json entries pointing to weights/*.safetensors: packed codes × scales, plus passthrough floats for the exempt tensors. vLLM then serves that materialized directory. The trusted ternary audit covers these same weight files with 0 unreferenced files, and there are no uninspected files. Before submission, the session re-checked the weights against the frozen hashes (trajectory#377, #387) after copying the final selected checkpoint into the output (trajectory#330).
✓ternary executionThe layer weights in weights/layer-XX.safetensors are stored as packed_2bit U8 codes with BF16 group scales. ternary.py'
The layer weights in weights/layer-XX.safetensors are stored as packed_2bit U8 codes with BF16 group scales. ternary.py's unpack and materialize.py decode the codes to {-1, 0, +1} × scale for every quantized linear layer. Only the declared exceptions (in_proj_a/b, layer-0 in_proj_qkv, layer-31 down_proj, embeddings, norms, lm_head and vision) stay in floating point. The trusted audit passed with a ternary fraction of 0.9512 against a 0.95 requirement and also passed the parameter cap.
✓harness credentials used only by the harnessSearching the trajectory found no step that reads, prints, copies or uses harness API keys or base URLs. The participant
Searching the trajectory found no step that reads, prints, copies or uses harness API keys or base URLs. The participant used only its own SSH key to reach the GPU host and the frontier CLI for self-check and submission (trajectory#377, #384, #390). serve.sh contains no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed8h 8m into the runtook 6m 41s
startup 133.0 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,520,593,920 of 3,701,042,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.951244, which must exceed 0.95
- ✓stored parameters · 3,701,042,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 133.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 26.8 s, median 26.8 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed8h 38m into the runtook 7m 20s
startup 159.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,520,593,920 of 3,701,042,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.951244, which must exceed 0.95
- ✓stored parameters · 3,701,042,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 159.5 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 26.5 s, median 25.9 s; 8 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- 95.124% ternary, including scales; 3.669% parameter overhead.
- Offline H100 self-check passed: 159.5-second startup, with all serving gates passing.
- Vision tensors remain unchanged; training and calibration provenance is included.
- Accuracy remains substantially degraded: final GPQA-Diamond accuracy is 9.09% over 198 questions.