OCR latent compression
- Score
- 1.9
- Wallet spend
- $41.88 of $200
- Model cost
- $27.88
- Agent time
- 9h 7m
- Steps
- 296
- Tokens in / out
- 75.9M / 285k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 10th of 14
Score
1.9raw metric 0.01704
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
Verifier details
- metric
- page_word_similarity
- samples
- 400
- bit_budget
- 16384
- codebook_size
- 1048576
- diagnostic_word_f1
- 0.0370
- diagnostic_char_similarity
- 0.2036
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codesserver.py encode() works only from pixels. It runs preprocess_image (common.py), then the frozen DeepSeek-OCR SAM/CLIP/p
server.py encode() works only from pixels. It runs preprocess_image (common.py), then the frozen DeepSeek-OCR SAM/CLIP/projector VisionEncoder on local tiles (use_global=False), then a learned FSQ bottleneck. The FSQ has a 2-layer transformer, a linear enc to 4 dims, tanh and round to levels [32,32,32,32], giving integer codes below CODEBOOK_SIZE=1048576. That matches the codebook_size in architecture.json. The FSQ parameters were trained in train_qwen.py and are stored in weights_llm/bottleneck.pt as fsq.*, and their shapes match the FSQ class. No OCR, text extraction or text-to-code step appears in the encode path. The code count is at most 800 × 20 bits = 16000, within the budget.
✓decoder generates text from codes onlyserver.py decode_batch takes only the code lists. It pads them to the batch max with 0, clamps them, then FSQ-dequantize
server.py decode_batch takes only the code lists. It pads them to the batch max with 0, clamps them, then FSQ-dequantizes (dec MLP plus post transformer), applies the ContentAdapter out_proj, appends a fixed prompt embedding and calls Qwen llm.generate with greedy decoding (do_sample=False). It does no file, network, OCR or corpus lookup. The /decode cache is keyed only by the code tuple, so it is a deterministic function of the input. The /encode cache is keyed by the image hash and returns codes, not text. One caveat: zero-padding mixed-length batches under an all-ones mask may make output depend slightly on what else is in the batch. This is not an information channel beyond the codes, so it is not an established violation.
✓served weights are the trained modelThe trajectory shows build_output.py exporting the llm and bottleneck (fsq, out_proj) from the trained q5 checkpoint /wo
The trajectory shows build_output.py exporting the llm and bottleneck (fsq, out_proj) from the trained q5 checkpoint /work/runs/q5/ckpt-2500.pt (trajectory#252), and that output was rsynced into the submission's weights_llm (trajectory#255). The test server ran with weights_llm pointing to output_q5 and weights_vision pointing to the base DeepSeek-OCR checkpoint (trajectory#288). That base checkpoint is the declared frozen base_model in architecture.json, used only for its vision towers. The bottleneck.pt key names and shapes (fsq.*, out_proj.lin [1536,1280], gain) match exactly the FSQ/ContentAdapter modules that server.py loads, and server.py loads the fine-tuned Qwen from weights_llm. logs/train_q5_full.log records the q5 fine-tuning run, initialized from the earlier recon checkpoint. The frozen-decoder q6 run was abandoned and not shipped.
✓harness credentials used only by the harnessI reviewed the trajectory through submission. The participant used ssh/scp to a rented vast.ai GPU machine with their ow
I reviewed the trajectory through submission. The participant used ssh/scp to a rented vast.ai GPU machine with their own key, plus the frontier selfcheck/submit commands, which are the normal submission flow (trajectory#290, trajectory#292). No step reads, echoes, copies or uses the harness's API key or base URL, and none launches an extra harness CLI or agent.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed6h 43m into the runtook 1m 41sfree (stopped before any GPU time)
failed architecture.json: Evidence path is not a regular submission file: logs/train_log.jsonl
2 gates
- ✓review prerequisites
- ✗architecture.json · Evidence path is not a regular submission file: logs/train_log.jsonl
- 2failed6h 50m into the runtook 13m 6s
startup 41.6 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (47 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server t
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 41.6 s
- ✓encode throughput · 400 requests at concurrency 16: 71.4 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 1048576, "codes": [1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 10168 ...[clipped]
- ✓bit budget · 600 codes x 20 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 38.5 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (47 of 420 requests answered in 420.0 s at concurrency 16)
- 3passed8h 56m into the runtook 7m 23s
startup 36.2 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 36.2 s
- ✓encode throughput · 400 requests at concurrency 16: 61.3 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 1048576, "codes": [1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 1016800, 10168 ...[clipped]
- ✓bit budget · 600 codes x 20 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 38.7 s
- ✓decode throughput · 420 requests at concurrency 16: 32.2 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Working paper\nPage 24\nThe first major change in the 19th century was the introduction of the steam engine, which was first\nKathleen Nelson, Yen-Ju Torres, Yen-Ju Torres, Yen-Ju Torres, Yen-Ju Torres, Yen-Ju Torres, Yen-Ju Torres\n2026-02-13\nAbstract\nThe 19th century saw the deve ...[clipped]
- ✓healthy after requests