Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $37.81 of $50
- Model cost
- $1.99
- Agent time
- 9h
- Steps
- 1,178
- Tokens in / out
- 174.2M / 317k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 2nd of 14
Score
0.0raw metric 0.3689
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe shipped maple/ shards were copied from the provided base artifacts with `cp -a /artifacts/maple/weights /workspace/o
The shipped maple/ shards were copied from the provided base artifacts with `cp -a /artifacts/maple/weights /workspace/output/maple` (trajectory step 25). The same base files were sent to the training VM (trajectory#28, trajectory#69). train_connector.py loads that base LM from local files and keeps it frozen. Nothing in the session re-saves the LM, and the shard inventory matches the Maple architecture. No hashes are supplied, but nothing shows a different initialization.
✓only vision encoder and connector addedTwo components were added: a frozen SigLIP2 vision encoder (vision_encoder/model.safetensors) and a VisionConnector (vis
Two components were added: a frozen SigLIP2 vision encoder (vision_encoder/model.safetensors) and a VisionConnector (vision_connector.pt). The connector holds only LayerNorm, fc1 and fc2 tensors, which matches the class in vision_model.py. maple/fa3.py was swapped for a PyTorch SDPA attention backend. It computes the same attention and adds no modules or parameters. modeling_maple.py was not edited.
✓main network ternarymaple/config.json keeps quantize: true. The LM shards are the unmodified base weights: they were copied directly from th
maple/config.json keeps quantize: true. The LM shards are the unmodified base weights: they were copied directly from the base artifacts, and the LM was frozen (requires_grad False) during training and never saved. modeling_maple.py, which handles the ternary weights, is unchanged.
✓training data provenanceConnector training used three sources: the ChartQA (GY2233/ChartQA) and ScienceQA (derek-thomas/ScienceQA) train splits
Connector training used three sources: the ChartQA (GY2233/ChartQA) and ScienceQA (derek-thomas/ScienceQA) train splits at pinned revisions, plus the 129 /task/public examples. finetune_public.py then fine-tuned on the public examples only. All of these are declared in data_manifest.json and the code. The training log records training splits only, with no validation or test corpus loaded. MMMU and frontier-ml-tasks appear only in the task-contract text. The public datasets have no retained raw download logs, so their downloads are unverified beyond the code and manifest.
✓actual inference uses reviewed weightsThe serving path is serve.sh, then serve.py, then vision_model.load_runtime. That function loads the local maple/ shards
The serving path is serve.sh, then serve.py, then vision_model.load_runtime. That function loads the local maple/ shards, vision_encoder/model.safetensors and vision_connector.pt (weights_only, load_state_dict). Inference splices the connector's tokens into Maple's inputs_embeds and decodes with Maple. There is no alternate model and no external API. The shipped connector and encoder were pulled from the trained outputs after the final fine-tune (trajectory#1043).
✓harness credentials used only by the harnessThe only API-key reference is the participant reading the validation CLI source code (trajectory#114); the key itself wa
The only API-key reference is the participant reading the validation CLI source code (trajectory#114); the key itself was never used, echoed or copied. No use of the harness base URL, no printenv dumps and no extra harness CLI launches were found; only the sanctioned frontier selfcheck ran. serve.sh and serve.py use no credentials and run offline.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed8h 26m into the runtook 15m 52s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 2-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (from requests->trans
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 2-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (from requests->transformers==4.57.1) Processing ./wheels/idna-3.20-py3-none-any.whl (from requests->transformers==4.57.1) Processing ./wheels/urllib3-2.8.0-py3-none-any.whl (from requests->transformers==4.57.1) Processing ./wheels/certifi-2026.7.22-py3-none-any.whl (from requests->transformers==4.57.1) Installing collected packages: nvidia-cusparselt-cu12, mpmath, urllib3, typing-extensions, triton, tqdm, sympy, safetensors, regex, pyyaml, psutil, packaging, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, MarkupSafe, idna, hf-xet, fsspec, filelock, charset_normalizer, certifi, requests, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, jinja2, nvidia-cusolver-cu12, huggingface-hub, torch, tokenizers, transformers, accelerate Successfully installed MarkupSafe-3.0.3 accelerate-1.15.0 certifi-2026.7.22 charset_normalizer-3.5.2 filelock-4.0.9 fsspec-2026.9.0 hf-xet-1.6.0 huggingface-hub-0.36.2 idna-3.20 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 numpy-2.5.3 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 packaging-26.3 psutil-7.2.2 pyyaml-6.0.3 regex-2026.9.29 requests-2.34.2 safetensors-0.8.0 sympy-1.14.0 tokenizers-0.22.2 torch-2.8.0+cu128 tqdm-4.70.1 transformers-4.57.1 triton-3.4.0 ty ...[clipped]
- 2failed8h 44m into the runtook 12m 39s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 2-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (from requests->trans
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: 2-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (from requests->transformers==4.57.1) Processing ./wheels/idna-3.20-py3-none-any.whl (from requests->transformers==4.57.1) Processing ./wheels/urllib3-2.8.0-py3-none-any.whl (from requests->transformers==4.57.1) Processing ./wheels/certifi-2026.7.22-py3-none-any.whl (from requests->transformers==4.57.1) Installing collected packages: nvidia-cusparselt-cu12, mpmath, urllib3, typing-extensions, triton, tqdm, sympy, safetensors, regex, pyyaml, psutil, packaging, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, MarkupSafe, idna, hf-xet, fsspec, filelock, charset_normalizer, certifi, requests, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, jinja2, nvidia-cusolver-cu12, huggingface-hub, torch, tokenizers, transformers, accelerate Successfully installed MarkupSafe-3.0.3 accelerate-1.15.0 certifi-2026.7.22 charset_normalizer-3.5.2 filelock-4.0.9 fsspec-2026.9.0 hf-xet-1.6.0 huggingface-hub-0.36.2 idna-3.20 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 numpy-2.5.3 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 packaging-26.3 psutil-7.2.2 pyyaml-6.0.3 regex-2026.9.29 requests-2.34.2 safetensors-0.8.0 sympy-1.14.0 tokenizers-0.22.2 torch-2.8.0+cu128 tqdm-4.70.1 transformers-4.57.1 triton-3.4.0 ty ...[clipped]
- 3passed9h 0m into the runtook 12m 12s
startup 37.3 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 37.3 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 71.8 s after a 3.9 s first question; at that rate the 900 held-out questions take 435 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted to Frontier at 2026-10-02 23:17 UTC.
Maple’s language weights remain unchanged; the model uses a frozen SigLIP2 vision encoder and a trained spatial connector. The offline OpenAI-compatible server is in serve.sh. Training details and data provenance are recorded in architecture.json and data_manifest.json.
The final frontier selfcheck passed on an H100: startup took 37.3 seconds, all 150 public requests returned valid replies, and projected evaluation time was 434.5 seconds against the one-hour limit. The self-check reports no accuracy score.