OCR latent compression
- Score
- 11.0
- Wallet spend
- $57.12 of $200
- Model cost
- $2.87
- Agent time
- 9h 34m
- Steps
- 1,045
- Tokens in / out
- 106.5M / 1.3M
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 9th of 19
Score
11.0raw metric 0.09781
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
Verifier details
- metric
- page_word_similarity
- samples
- 400
- bit_budget
- 16384
- codebook_size
- 65536
- diagnostic_word_f1
- 0.2217
- diagnostic_char_similarity
- 0.3117
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codesIn server.py, /encode calls modeling.encode_batch, which is purely visual. The frozen DeepSeek-OCR SAM and CLIP towers a
In server.py, /encode calls modeling.encode_batch, which is purely visual. The frozen DeepSeek-OCR SAM and CLIP towers and the projector produce 1280-d cell features. These pass through the neck, which is frozen and acts as an identity, and are then quantized by argmin against a learned VQ codebook of 65536×1280. The codebook is trained in train.py with a straight-through estimator and learn_codebook=1, and is shipped as vq.embed.weight in weights/page_tokenizer.safetensors. Each image gives 832 codes × 16 bits = 13312 bits. There is no OCR, text extraction or text side channel anywhere in the encode path.
✓decoder generates text from codes onlyIn server.py, /decode validates the integer codes and maps them through vq.decode, the codebook embedding lookup in mode
In server.py, /decode validates the integer codes and maps them through vq.decode, the codebook embedding lookup in modeling.code_sequence. It adds the fixed image_newline / view_seperator layout tokens and the constant prompt "\nFree OCR.", then decodes greedily with the fine-tuned DeepSeek-V2 LM via fastgen.FastDecoder. The static KV cache is overwritten on every prefill. There is no per-image persistent state, transcript store, retrieval, file or network lookup, so the only per-image information is the codes.
✓served weights are the trained modelserve.sh execs server.py, which loads weights/page_tokenizer.safetensors (420 tensors) on top of the declared base, deep
serve.sh execs server.py, which loads weights/page_tokenizer.safetensors (420 tensors) on top of the declared base, deepseek-ai/DeepSeek-OCR at revision 9f30c71f. The meta file records source ckpt_2750.pt, step 2750, run8, learn_codebook=1 and freeze_neck=1. The tensor inventory matches what export_weights.py writes from the train.py checkpoints: trainable LM layers, image_newline, vision params and the fp32 VQ buffers. Trajectory step 1013 shows finalize.sh exporting /home/dev/ckpt_keep/ckpt_2750.pt with export_weights.py and copying the result to weights/page_tokenizer.safetensors. In step 1014 the md5 of the shipped file matches the exported file, and the meta reports ckpt_2750.pt, step 2750, 420 tensors. logs/train8.log, fetched in step 1015, covers that run.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC, OPENROUTER, API_KEY, BASE_URL, claude -p, anthropic, openai, codex, chat/comple
I searched the trajectory for ANTHROPIC, OPENROUTER, API_KEY, BASE_URL, claude -p, anthropic, openai, codex, chat/completions, printenv and curl. None of the hits shows the participant reading or using harness credentials. The curl calls only check connectivity to huggingface.co or pypi, check the health of the local test server, or download base weights from HF. The only credential-related text found was Spin tooling documentation in the early steps.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed3h 40m into the runtook 22sfree (stopped before any GPU time)
failed architecture.json: Evidence path is not a regular submission file: logs/train_log.jsonl
2 gates
- ✓review prerequisites
- ✗architecture.json · Evidence path is not a regular submission file: logs/train_log.jsonl
- 2failed3h 41m into the runtook 5m 59s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: no DeepSeek-OCR weights found in ['/submission/weights/base', '/artifacts/deepseek-ocr/weights', '
3 gates
- ✓review prerequisites
- ✓architecture.json
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: no DeepSeek-OCR weights found in ['/submission/weights/base', '/artifacts/deepseek-ocr/weights', '/submission/weights/base']
- 3failed3h 48m into the runtook 10m 32s
startup 13.8 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (177 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 13.8 s
- ✓encode throughput · 400 requests at concurrency 16: 23.9 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 65536, "codes": [64303, 14306, 44417, 13994, 13994, 8798, 55451, 56601, 7216, 51254, 57436, 49501, 34307, 24133, 61910, 32830, 55324, 5815, 30929, 13205, 48740, 18025, 18025, 18025, 20451, 20301, 41277, 24700, 24700, 24700, 24700, 24700, 24700, 24700, 24700, 24700, 2 ...[clipped]
- ✓bit budget · 832 codes x 16 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 13.2 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (177 of 420 requests answered in 420.0 s at concurrency 16)
- 4passed4h 41m into the runtook 6m 28s
startup 12.9 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 12.9 s
- ✓encode throughput · 400 requests at concurrency 16: 23.9 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 65536, "codes": [64303, 14306, 44417, 13994, 13994, 8798, 55451, 56601, 7216, 51254, 57436, 49501, 34307, 24133, 61910, 32830, 55324, 5815, 30929, 13205, 48740, 18025, 18025, 18025, 20451, 20301, 41277, 24700, 24700, 24700, 24700, 24700, 24700, 24700, 24700, 24700, 2 ...[clipped]
- ✓bit budget · 832 codes x 16 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 10.5 s
- ✓decode throughput · 420 requests at concurrency 16: 178.3 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Yury Gjini, Yosef Boronat, Alexander Penchala, Jacob Shahrampour\nNina, Salva Wagner\nJul 25, 2024\nAbstract\nGeneralized linear models are a class of regression models that are used to model response variables that are not normally distributed . The development of the new system was ...[clipped]
- ✓healthy after requests
- 5passed9h 23m into the runtook 8m 41s
startup 10.4 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 10.4 s
- ✓encode throughput · 400 requests at concurrency 16: 22.8 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 65536, "codes": [22914, 24880, 774, 41864, 41864, 455, 62941, 25326, 14559, 46803, 64670, 56235, 52225, 51161, 6952, 25005, 34618, 15170, 45951, 20836, 4293, 52067, 29193, 40668, 65400, 22820, 9071, 46912, 12232, 8479, 59974, 52764, 55837, 5395, 11569, 50980, 9360, 3 ...[clipped]
- ✓bit budget · 832 codes x 16 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 13.9 s
- ✓decode throughput · 420 requests at concurrency 16: 176.2 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Mahdi Gjini, Youlu Boronat, Alexander Penchala, Jacob Shahrampour, Ivan\nNiu, Rania Milling\nJul 4, 2026\nAbstract\nGeneral strike against forces that responsibly took radio equipment earlier violated movement that Comintern demanded .\nCurrent proposals are made off methodologically ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Final state of the work:
What was built
A discrete visual tokenizer for document pages and screenshots, in /workspace/output/:
Encoder (no transcription, no text tokens): frozen DeepSeek-OCR vision tower (SAM ViT-B + CLIP ViT-L) → frozen 2-layer transformer neck → vector quantiser over the page's cells. 24×24 local cells (64 px) + 16×16 global cells (64 px) + a view separator = 832 codes × 16 bits = 13,312 bits (under the 16,384-bit budget). The code book has 65,536 entries of 1,280 dims.
Decoder: DeepSeek-V2-Lite MoE LM (12 layers, 64 routed + 2 shared experts) reading the looked-up code vectors, served through a dense-expert path with a captured CUDA graph and a fixed 16-row batch (so the same code list always decodes to the same text).
Training: the code book is shaped by the language loss alone (no reconstruction loss, EMA off, k-means init, neck residual off and frozen); the LM is fine-tuned beside it at 3e-5. Chain: run2 → run5 → run6 (abandoned) → run8, which shipped run8/ckpt_2750.
Key numbers
- 0.3471 word-level score on 64 held-out synthetic development pages (0.3742 on the 32-page set) vs 0.3152 for the previous run's checkpoint.
- Fixing a real serving bug found late — the frozen neck's residual is not in a checkpoint, so the server was re-adding a random residual the code book had never seen — was worth ~+0.015.
- 13,312 bits/image; code book uses 26,172 of 65,536 entries (14.26 bits entropy/code).
- Final selfcheck passed: startup 10.4 s/300 s, encode 17.5 img/s, decode 2.38 img/s (420 requests in 176 s of a 420 s deadline), bit budget and response-shape checks green.
No public validation or test split was used for training, calibration or selection — all development pages are rendered from the wikitext-103 train split with seeds the training stream never reaches (documented in data_manifest.json, logs/select.json).