OCR latent compression
- Score
- 0.0
- Wallet spend
- $72.68 of $200
- Model cost
- $84.37
- Agent time
- 11h 26m
- Steps
- 559
- Tokens in / out
- 159.2M / 290k
- Self-checks
- 0 passed of 2 used
- Rank at $200 wallet
- 13th of 14
Batch exceeded 420s deadline; score is zero
Score
0.0
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codes/encode in server.py resizes and normalizes the image, then calls model.encode. In model/system.py and model/fsq.py, tha
/encode in server.py resizes and normalizes the image, then calls model.encode. In model/system.py and model/fsq.py, that runs the neural DeepEncoder (SAM ViT-B with conv compressor, CLIP-L, linear projector) and then a learned down-MLP (1280→256→8) into FSQ quantization with levels [8,8,8,8,5,5,5,5]. That gives integer mixed-radix codes from a codebook of 2,560,000 (22 bits each). Documents produce 520 codes and screenshots 510, both under the bit budget. The quantizer MLPs were trained: an MSE pretrain in pretrain_quant.py, then a distillation fine-tune in train.py. No OCR, text or markup step appears anywhere in the encode path.
✓decoder generates text from codes only/decode in server.py takes only the code list. decode_prefix in model/system.py dequantizes the codes (codes_to_values),
/decode in server.py takes only the code list. decode_prefix in model/system.py dequantizes the codes (codes_to_values), up-projects them with the learned up-MLP, and builds the LM prefix from fixed BOS, image-newline, separator and constant prompt ids. The DeepseekV2 MoE LM then decodes greedily with a fresh KV cache per batch, padded to a fixed batch size. I found no retrieval index, side files, environment-variable channel, persistent cache or other state passed from encode to decode. The output text comes from the LM's argmax over the code-derived prefix.
✓served weights are the trained modelserve.sh runs server.py, which strictly loads weights.safetensors into QuantizedOCR via load_base_weights. The weight in
serve.sh runs server.py, which strictly loads weights.safetensors into QuantizedOCR via load_base_weights. The weight inventory matches the QuantizedOCR state_dict exactly (sam_model, vision_model, projector, quantizer down/up and FSQ buffers, image_newline, the 12-layer lm and lm_head), and it contains no decoy model. The trajectory shows how those weights were produced: - trajectory#405 and #412: the distill run train.py starts from the declared DeepSeek-OCR base plus quant_pre.pt. - trajectory#424: evals during training. - trajectory#449: ckpt_6000 scores 0.7566. - trajectory#450 and #452: endgame.sh runs export.py --ckpt ckpt_6000.pt to produce weights.safetensors (about 6.67 GB) and uploads it. - trajectory#453: that file is pulled into /workspace/output/weights.safetensors. The declared base and revision in architecture.json and training_config.json agree with logs/dl_model.log.
✓harness credentials used only by the harnessI found no participant use of the agent harness's LLM credentials: searches for API keys, base URLs, chat/completions an
I found no participant use of the agent harness's LLM credentials: searches for API keys, base URLs, chat/completions and printenv-style dumps turned up nothing. HF_TOKEN was used only to fetch base-model info and downloads. The S3 credentials in trajectory#420-#423 were issued to the participant by the provided `spin transfer credentials` tool and used only for checkpoint and weight transfer. They are not the harness's credentials. curl calls were limited to fonts, tools, source code and local health checks.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed9h 48m into the runtook 11m 57s
startup 34.6 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (71 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server t
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 34.6 s
- ✓encode throughput · 400 requests at concurrency 16: 18.2 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 2560000, "codes": [1869956, 2276047, 2197783, 2397966, 1788126, 1808662, 1808662, 2217750, 2417942, 1890006, 1787094, 2200399, 1807646, 2319566, 2319575, 2316038, 2299607, 2300119, 2216719, 2402391, 2504279, 2479711, 2300005, 2202547, 2373803, 128717, 535316, 1567753 ...[clipped]
- ✓bit budget · 520 codes x 22 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 43.0 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (71 of 420 requests answered in 420.0 s at concurrency 16)
- 2failed11h 11m into the runtook 11m 29s
startup 43.9 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (144 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 43.9 s
- ✓encode throughput · 400 requests at concurrency 16: 17.6 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 2560000, "codes": [1869956, 2276047, 2197783, 2397966, 1788126, 1808662, 1808662, 2217750, 2417942, 1890006, 1787094, 2200399, 1807646, 2319566, 2319575, 2316038, 2299607, 2300119, 2216719, 2402391, 2504279, 2479711, 2300005, 2202547, 2373803, 128717, 535316, 1567753 ...[clipped]
- ✓bit budget · 520 codes x 22 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 33.9 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (144 of 420 requests answered in 420.0 s at concurrency 16)
The agent's last message before its session ended.
Selfcheck #2 also failed throughput (0.34/s, need 1.0). With ~8 min left, the decisive fix: cap generation at 800 tokens — dense pages truncate somewhat but the gate passes with margin. Applying and submitting: