OCR latent compression
- Score
- 67.8
- Wallet spend
- $76.49 of $200
- Model cost
- $183.84
- Agent time
- 7h 54m
- Steps
- 301
- Tokens in / out
- 81.9M / 462k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
67.8raw metric 0.6055
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
Verifier details
- metric
- page_word_similarity
- samples
- 400
- bit_budget
- 16384
- codebook_size
- 16
- diagnostic_word_f1
- 0.6869
- diagnostic_char_similarity
- 0.7050
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codesserver.py's do_encode runs preprocess_image, then Encoder.encode in encoder.py. That function resizes the image determin
server.py's do_encode runs preprocess_image, then Encoder.encode in encoder.py. That function resizes the image deterministically to one of two declared grids, normalizes it, and runs the learned DeepSeek-OCR DeepEncoder: SAM-ViT-B, then CLIP-L, then concatenation, then the projector. The features then go through a learned FSQ bottleneck: tanh(bottleneck.down(feats)), rounded to 16 levels (round(z*7.5+7.5), clamped to 0..15). The output is [grid_id] + 252x16 codes. That is 4033 codes x 4 bits = 16132 bits, and codebook_size is 16, matching architecture.json and config.json. The training quantizer in train/qocr.py (FSQBottleneck with a straight-through estimator, trained end to end with text cross-entropy) is the same. No OCR, text extraction or text side channel appears anywhere in the encode path.
✓decoder generates text from codes onlyserver.py's embeds_from_codes dequantizes the integer codes with (c-HALF)/HALF and maps them through the learned up-MLP
server.py's embeds_from_codes dequantizes the integer codes with (c-HALF)/HALF and maps them through the learned up-MLP (bottleneck.up*) into vision embeddings. It inserts the learned image_newline and view_seperator tokens and wraps everything in fixed prompt token ids from config.json. The FastLM decoder in fastlm.py (the fine-tuned DeepSeek-3B-MoE) then generates text by greedy argmax, and the tokenizer detokenizes it. Each request's KV cache rows are written before they are read and masked by position, so no state carries over between requests. Decode uses no per-request file I/O, retrieval, caches, environment variables or access to the original image. The only input is the codes.
✓served weights are the trained modelserve.sh runs server.py, which loads model.safetensors. The Encoder (strict missing-key assertion), FastLM, Up, image_ne
serve.sh runs server.py, which loads model.safetensors. The Encoder (strict missing-key assertion), FastLM, Up, image_newline and view_seperator all come from that single state dict. The weight inventory contains exactly these keys (bottleneck.down 16x1280, bottleneck.up.*, bottleneck.up_lin, LM layers, embed_tokens, lm_head, SAM and vision tensors) and no unused codec tensors, consistent with codec_layers 0. In trajectory#291, avg_ckpt.py averaged the 5 trained checkpoints listed in train/FINAL_CHECKPOINT.txt and merge_ckpt.py merged them onto the declared base, deepseek-ai/DeepSeek-OCR at its pinned revision, giving sha256 c84e4ca0.... trajectory#292 copied that file to output/model.safetensors, and trajectory#294 checked that the shipped file's sha256 matches. The run configs (codec_layers 0, FSQ 16/16) and the resume chain (run1 -> run1b -> run8/run1c) agree with the served config and architecture.json. A later 6-checkpoint average was evaluated but never shipped.
✓harness credentials used only by the harnessSearching the trajectory for credential patterns (ANTHROPIC, claude, codex, printenv, API_KEY, BASE_URL, openrouter) fou
Searching the trajectory for credential patterns (ANTHROPIC, claude, codex, printenv, API_KEY, BASE_URL, openrouter) found matches only in the Spin CLI documentation in the initial prompt (trajectory/steps-1-17). The participant never read, printed, copied or used harness credentials, and never launched extra model or CLI sessions; the single Agent tool call was the harness's own subagent. HF_TOKEN was used only for model downloads on the compute box, and the session otherwise used normal Spin compute and storage commands, such as the s5cmd copy in trajectory#292. Neither serve.sh nor server.py uses any credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed6h 49m into the runtook 7m 32s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: weights loaded 0.09351515769958496 Traceback (most recent call last): File "/submission/server.p
3 gates
- ✓review prerequisites
- ✓architecture.json
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: weights loaded 0.09351515769958496 Traceback (most recent call last): File "/submission/server.py", line 20, in <module> from encoder import Encoder, preprocess_image, grid_shape, set_grids File "/submission/encoder.py", line 11, in <module> from codec import Codec ModuleNotFoundError: No module named 'codec'
- 2passed6h 57m into the runtook 4m 34s
startup 8.0 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 8.0 s
- ✓encode throughput · 400 requests at concurrency 16: 16.0 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16, "codes": [0, 6, 5, 5, 6, 7, 6, 9, 8, 8, 7, 8, 7, 7, 6, 7, 7, 6, 1, 4, 8, 4, 6, 7, 7, 8, 10, 9, 8, 6, 5, 8, 9, 6, 1, 6, 6, 0, 7, 6, 5, 7, 13, 10, 9, 2, 5, 10, 9, 3, 2, 11, 5, 2, 4, 9, 7, 6, 7, 7, 9, 6, 8, 8, 8, 5, 4, 10, 4, 4, 3, 9, 5, 6, 6, 4, 8, 6, 8, 7, 8, 3, 5 ...[clipped]
- ✓bit budget · 4033 codes x 4 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 6.5 s
- ✓decode throughput · 420 requests at concurrency 16: 76.0 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Emotion-driven Facial Generation Using Geometric Constraints for Natural and Realistic Talking Faces\nMahdi Gijon, Younji Boronat, Alexander Penchkala, Jacob Shahnazaryan, Yeni Ju\nNiu, Lisa Millling\nJul 2, 2026\nAbstract\nGenerating talking faces that respond naturally to audio inp ...[clipped]
- ✓healthy after requests
- 3passed7h 40m into the runtook 15m 14s
startup 8.0 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 8.0 s
- ✓encode throughput · 400 requests at concurrency 16: 14.8 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16, "codes": [0, 5, 5, 5, 7, 7, 6, 10, 8, 9, 8, 7, 7, 7, 7, 7, 8, 6, 5, 5, 7, 7, 6, 9, 8, 8, 8, 8, 7, 7, 7, 7, 8, 6, 5, 5, 7, 7, 6, 9, 9, 8, 8, 8, 8, 7, 7, 7, 7, 6, 5, 6, 6, 6, 7, 9, 9, 8, 7, 8, 8, 8, 6, 7, 7, 6, 5, 5, 7, 6, 6, 9, 8, 8, 8, 8, 8, 7, 6, 7, 8, 7, 1, 4, ...[clipped]
- ✓bit budget · 4033 codes x 4 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 8.3 s
- ✓decode throughput · 420 requests at concurrency 16: 80.3 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Emotion-driven Facial Generation Using Geometric Constraints for Natural and Realistic Talking Faces\nMahdi Gijon, Yuqor Boronat, Alexander Penchala, Jacob Shahnazaryan, Yen Jua\nNiu, Lisa Millang\nJul 2, 2026\nAbstract\nGenerating talking faces that respond naturally to audio input ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Checkpoint averaging helps: the average of run 1b's step 7000 and 8000 scores 0.565, above either alone. Next I'll push run 1b's step-10000 checkpoint and evaluate run 7's final checkpoint on box 3.