Ternary quantization
- Score
- 0.0
- Wallet spend
- $40.04 of $50
- Model cost
- $64.39
- Agent time
- 11h 25m
- Steps
- 634
- Tokens in / out
- 191.8M / 354k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 11th of 14
Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh 16:16:34] installing wheels [serve.sh 16:17:51] dequantizing Traceback (most recent call la
Score
0.0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startAt trajectory step 39 the participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13cc
At trajectory step 39 the participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /root/work/qwen3.5-4b. evidence/quantize.py loads language_model weights from both shards in that directory and asserts there are no missing or unexpected keys. It then rotates, ternary-GPTQ quantizes and writes the manifest with the pinned base_revision. qat_sft.py and pack_qat.py continue from that quantized artifact. architecture.json declares the same base and revision. The trusted ternary audit inventory has 4,659,865,088 parameters, which matches the base model's metadata.
✓text only inferenceserve.sh runs code/build_serve_dir.py and then serves the output directory with the vLLM OpenAI server. code/dequant_lib
serve.sh runs code/build_serve_dir.py and then serves the output directory with the vLLM OpenAI server. code/dequant_lib.build_state_dict keeps only model.language_model.* and lm_head.weight, so the vision and MTP tensors are dropped. build_serve_dir writes a config taken from text_config with architectures ["Qwen3_5ForCausalLM"], which is a text-only causal LM. No image or video processing path is served.
✓no public benchmark eval trainingAll declared training, calibration and distillation corpora use train-type splits. These are fineweb sample-10BT train,
All declared training, calibration and distillation corpora use train-type splits. These are fineweb sample-10BT train, ultrachat_200k train_sft, OpenR1-Math-220k train, gsm8k train, hendrycks_math algebra train, ai2_arc train, cais/mmlu auxiliary_train and Magicoder-OSS-Instruct-75K train. They are declared in evidence/prep_data.py, evidence/gen_qat_data.py and the *.pt.manifest.json files, and the trajectory confirms them. GPQA and LiveCodeBench dev data were used only for evaluation through validate.py, and no test-split data was found in training. The sources are declared but no download log was retained, so they are recorded as unverified.
✓actual inference uses reviewed weightsThe serving call graph is serve.sh, then build_serve_dir.py, then dequant_lib.build_state_dict. That code reads quantiza
The serving call graph is serve.sh, then build_serve_dir.py, then dequant_lib.build_state_dict. That code reads quantization_manifest.json and only the shipped weights/lang-codes, lang-scales and lang-float safetensors. It saves them as the model.safetensors that vLLM loads from $SCRATCH/deq. No other weight source or download appears in the serving path. The trusted audit reports 0 unreferenced parameters, and its parameter-cap check passed.
✓ternary executionAll attention, linear-attention and MLP projection weights are stored as packed_2bit codes in {-1, 0, +1}, with fp32 sca
All attention, linear-attention and MLP projection weights are stored as packed_2bit codes in {-1, 0, +1}, with fp32 scales per (row, 64-column group). dequant_lib multiplies the decoded codes by their scales to produce the served weights. Any encoding other than float or packed_2bit raises an error. The float tensors are limited to embed, lm_head, norms, conv1d, A_log/dt_bias and in_proj_a/b, and lang-float contains no mlp or projection tensors. The trusted ternary audit passed with a ternary fraction of 0.983.
✓harness credentials used only by the harnessThe trajectory shows no participant use of the harness's API key or base URL, no printing or copying of environment vari
The trajectory shows no participant use of the harness's API key or base URL, no printing or copying of environment variables, and no extra agent or CLI sessions. Teacher generation used httpx against a vLLM server running locally on the compute box. HF_TOKEN appears only in Hugging Face download commands (e.g. step 39), which is a download credential, not a harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed4h 19m into the runtook 7m 47s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [--watermark-config WATERMARK_CONFIG] [--diffusion-config DIFF
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [--watermark-config WATERMARK_CONFIG] [--diffusion-config DIFFUSION_CONFIG] [--kv-transfer-config KV_TRANSFER_CONFIG] [--kv-events-config KV_EVENTS_CONFIG] [--ec-transfer-config EC_TRANSFER_CONFIG] [--ec-manager-config EC_MANAGER_CONFIG] [--compilation-config COMPILATION_CONFIG] [--attention-config ATTENTION_CONFIG] [--engram-config ENGRAM_CONFIG] [--reasoning-config REASONING_CONFIG] [--kernel-config KERNEL_CONFIG] [--additional-config ADDITIONAL_CONFIG] [--structured-outputs-config STRUCTURED_OUTPUTS_CONFIG] [--aux-output-config AUX_OUTPUT_CONFIG] [--profiler-config PROFILER_CONFIG] [--optimization-level OPTIMIZATION_LEVEL] [--performance-mode {balanced,interactivity,throughput}] [--weight-transfer-config WEIGHT_TRANSFER_CONFIG] [--disable-log-stats] [--aggregate-engine-logging] [--fail-on-environ-validation | --no-fail-on-environ-validation] [--shutdown-timeout SHUTDOWN_TIMEOUT] [--gdn-prefill-backend {flashinfer,triton,cutedsl}] [--kda-prefill-backend {auto,triton,flashkda,flashinfer,fused}] [--kda-decode-backend {auto,native,flashinfer,triton}] [--enable-log-requests | --no-enable-log-requests] [model_tag] api_server.py: error: argument --compilation-config/-cc: 1 validation error for CompilationConfig cudagraph_config Unexpected keyword ar ...[clipped]
- 2failed6h 49m into the runtook 7m 29s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: async with build_async_engine_client( File "/usr/lib/python3.12/contextlib.py", line 210, in
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: async with build_async_engine_client( File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/entrypoints/launchers/api_server/entry.py", line 58, in build_async_engine_client async with build_async_engine_client_from_engine_args( File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/entrypoints/launchers/api_server/entry.py", line 92, in build_async_engine_client_from_engine_args async_llm = AsyncLLM.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/v1/engine/async_llm.py", line 248, in from_vllm_config return cls( ^^^^ File "/tmp/pylibs/vllm/v1/engine/async_llm.py", line 180, in __init__ self.engine_core = EngineCoreClient.make_async_mp_client( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/v1/engine/core_client.py", line 167, in make_async_mp_client return AsyncMPClient( ^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/tracing/otel.py", line 175, in sync_wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/tmp/pylibs/vllm/v1/engine/core_client.py", line 1100, in __init__ super().__init__( File "/tmp/pylibs/vllm/v1/engine/core_client.py", line 674, in __init__ with launch_core_engines( File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/tmp/pylibs/vllm/v1/engine/utils.py", line 1268, in launch_core_engines wait_for_engine_startup( File "/t ...[clipped]
- 3failed8h 49m into the runtook 8m 42s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh 13:34:56] installing wheels [serve.sh 13:37:19] dequantizing dequantized 427 tensors ->
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh 13:34:56] installing wheels [serve.sh 13:37:19] dequantizing dequantized 427 tensors -> /tmp/deq/model.safetensors (6s) serve dir ready in 6s [serve.sh 13:37:27] starting vllm Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/tmp/pylibs/vllm/entrypoints/openai/api_server.py", line 5, in <module> from vllm.entrypoints.launchers.api_server.app_state import init_app_state File "/tmp/pylibs/vllm/entrypoints/launchers/api_server/app_state.py", line 9, in <module> from vllm.engine.protocol import EngineClient File "/tmp/pylibs/vllm/engine/protocol.py", line 23, in <module> from vllm.v1.engine.input_processor import InputProcessor File "/tmp/pylibs/vllm/v1/engine/input_processor.py", line 21, in <module> from vllm.multimodal.encoder_budget import MultiModalBudget File "/tmp/pylibs/vllm/multimodal/encoder_budget.py", line 15, in <module> from vllm.v1.core.encoder_cache_manager import compute_mm_encoder_budget File "/tmp/pylibs/vllm/v1/core/encoder_cache_manager.py", line 11, in <module> from vllm.v1.request import Request File "/tmp/pylibs/vllm/v1/request.py", line 25, in <module> from vllm.v1.structured_output.request import StructuredOutputRequest File "/tmp/pylibs/vllm/v1/structured_output/__init__.py", line 10, in <module> from vllm.parser.engine.adapters import ParserEngineReasoningAdapter File "/tmp/pylibs/vllm/parser/__init__.py", line 4, in <module> from vllm.parser.abstract_parser import ( File "/tmp/pylibs/vllm/parser/abstract_parser.py", line 12, in <module> from xgrammar import Grammar, StructuralTag File "/tmp/pylibs/xgrammar/__init__.py", line 1, in <module> from . import exception, load_binding, stru ...[clipped]
- 4failed11h 5m into the runtook 8m 6s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh 15:50:04] installing wheels [serve.sh 15:51:24] dequantizing Traceback (most recent call
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh 15:50:04] installing wheels [serve.sh 15:51:24] dequantizing Traceback (most recent call last): File "/submission/code/build_serve_dir.py", line 7, in <module> import torch File "/dev/shm/pylibs/torch/__init__.py", line 444, in <module> _load_global_deps() File "/dev/shm/pylibs/torch/__init__.py", line 402, in _load_global_deps _preload_cuda_deps(err) File "/dev/shm/pylibs/torch/__init__.py", line 358, in _preload_cuda_deps raise err File "/dev/shm/pylibs/torch/__init__.py", line 380, in _load_global_deps ctypes.CDLL(global_deps_lib_path, mode=ctypes.RTLD_GLOBAL) File "/usr/lib/python3.12/ctypes/__init__.py", line 379, in __init__ self._handle = _dlopen(self._name, mode) ^^^^^^^^^^^^^^^^^^^^^^^^^ OSError: /dev/shm/pylibs/torch/lib/libtorch_global_deps.so: failed to map segment from shared object
The agent's last message before its session ended.
Everything is validated and consistent. Submitting: