Ternary quantization
- Score
- 0.0
- Wallet spend
- $57.53 of $200
- Model cost
- $5.13
- Agent time
- 5h 45m
- Steps
- 435
- Tokens in / out
- 67.3M / 137k
- Self-checks
- 0 passed of 4 used
- Rank at $200 wallet
- 17th of 19
The reviewer found a rule violation; see the Review tab.
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✗actual inference uses reviewed weights
The serving path in inference.py rebuilds every packed linear weight as alpha*T + mu. It reads the per-block '<name>.offset' tensors and the '<name>.perm' tensors from weights/packed.safetensors and uses both as active inference parameters. The trusted ternary audit lists exactly these .offset and .perm tensors as unreferenced (28,780,544 parameters), and quantization_manifest.json references only the scale for each packed_2bit tensor. So the served weights depend on active inference parameters that the numerical audit never reviewed. This is not shared storage or an unused training artifact.
✗ternary execution
The policy requires every packed_2bit tensor to be served as code times scale. inference.py instead serves each one as scale*T + offset, with a per-block full-precision bf16 offset added that the manifest does not declare, and then column-permutes the result. The effective served matrix is therefore not the code×scale weight the manifest declares and the audit checked, even though the audit's ternary-fraction check passed. training_evidence.md itself states 'dequantized weight = scale * T + offset'.
✓pinned qwen startarchitecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The trajectory shows po
architecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The trajectory shows post-training quantization (PT2-LLM ATQ+SSR) run directly on the pinned /artifacts/qwen3.5-4b checkpoint, whose revision STARTING_ARTIFACT.json confirms. The packed checkpoint has the full Qwen3.5 language-model tensor structure. Only the visual and MTP tensors are left out, and the audit records them as omitted.
✓text only inferenceinference.py tokenizes chat messages with tokenizer.apply_chat_template and calls model.generate. It has no image proces
inference.py tokenizes chat messages with tokenizer.apply_chat_template and calls model.generate. It has no image processor, no multimodal inputs and no MTP use. No model.visual.* or mtp.* tensors are shipped in weights/packed.safetensors.
✓no public benchmark eval trainingCalibration used 32 samples of 512 tokens from the WikiText-2 (Salesforce/wikitext) train split, as declared in training
Calibration used 32 samples of 512 tokens from the WikiText-2 (Salesforce/wikitext) train split, as declared in training_evidence.md and quantization.log and shown in the trajectory. The PT2-LLM data prep also saved a WikiText test split, but it was used only for perplexity reporting. Nothing shows it was used to train, calibrate or select. No public benchmark eval data was used.
✓harness credentials used only by the harnessThe trajectory shows only spin s5cmd uploads to permitted wallet storage and Hugging Face dataset downloads. No harness
The trajectory shows only spin s5cmd uploads to permitted wallet storage and Hugging Face dataset downloads. No harness CLI launches appear, and no use of harness API keys or base URLs from the participant's own code was found (searches for ANTHROPIC and OPENROUTER returned no misuse).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed4h 17m into the runtook 41sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
- 2failed4h 17m into the runtook 18m 47s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/inference.py", line 2, in <module> impo
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,597,935,616 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.991983, which must exceed 0.95
- ✓stored parameters · 3,626,716,160 stored (codes, scales, retained floats and 28,780,544 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/inference.py", line 2, in <module> import torch ModuleNotFoundError: No module named 'torch'
- 3failed4h 36m into the runtook 12m 37s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Processing ./deps/filelock-4.2.4-py3-none-any.whl (from torch) Requirement already satisfied: typi
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,597,935,616 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.991983, which must exceed 0.95
- ✓stored parameters · 3,626,716,160 stored (codes, scales, retained floats and 28,780,544 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Processing ./deps/filelock-4.2.4-py3-none-any.whl (from torch) Requirement already satisfied: typing-extensions>=4.10.0 in /opt/venv/lib/python3.12/site-packages (from torch) (4.16.0) INFO: pip is looking at multiple versions of torch to determine which version is compatible with other requirements. This could take a while. ERROR: Could not find a version that satisfies the requirement setuptools<82 (from torch) (from versions: 84.2.1) ERROR: No matching distribution found for setuptools<82 Traceback (most recent call last): File "/submission/inference.py", line 2, in <module> import torch ModuleNotFoundError: No module named 'torch'
- 4failed4h 49m into the runtook 7m 41s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Processing ./deps/jinja2-3.1.6-py3-none-any.whl (from torch) Processing ./deps/fsspec-2026.9.0-py3
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,597,935,616 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.991983, which must exceed 0.95
- ✓stored parameters · 3,626,716,160 stored (codes, scales, retained floats and 28,780,544 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Processing ./deps/jinja2-3.1.6-py3-none-any.whl (from torch) Processing ./deps/fsspec-2026.9.0-py3-none-any.whl (from torch) INFO: pip is looking at multiple versions of torch to determine which version is compatible with other requirements. This could take a while. ERROR: Could not find a version that satisfies the requirement cuda-toolkit==12.8.1; platform_system == "Linux" (from torch) (from versions: 13.0.3.0) ERROR: No matching distribution found for cuda-toolkit==12.8.1; platform_system == "Linux" Traceback (most recent call last): File "/submission/inference.py", line 2, in <module> import torch ModuleNotFoundError: No module named 'torch'
- 5failed4h 58m into the runtook 6m 42s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: n. The above exception was the direct cause of the following exception: Traceback (most recent c
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,597,935,616 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.991983, which must exceed 0.95
- ✓stored parameters · 3,626,716,160 stored (codes, scales, retained floats and 28,780,544 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: n. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/submission/inference.py", line 148, in <module> main() File "/submission/inference.py", line 55, in main model = AutoModelForCausalLM.from_pretrained(model_dir, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 365, in from_pretrained explicit_local_code = has_local_code and not _get_model_class( ^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 183, in _get_model_class supported_models = model_mapping[type(config)] ~~~~~~~~~~~~~^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 611, in __getitem__ return self._load_attr_from_module(model_type, model_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 625, in _load_attr_from_module return getattribute_from_module(self._modules[module_name], attr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 537, in getattribute_from_module if hasattr(module, attr): ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/utils/import_utils.py", line 2688, in __getattr__ raise ModuleNotFoundError(import_error_messag ...[clipped]
The agent's last message before its session ended.
Submitted.