OCR latent compression
- Score
- 4.2
- Wallet spend
- $8.91 of $10
- Model cost
- $45.69
- Agent time
- 8h 7m
- Steps
- 604
- Tokens in / out
- 61.8M / 489k
- Self-checks
- 2 passed of 4 used
- Rank at $10 wallet
- 11th of 14
Score
4.2raw metric 0.03772
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
Verifier details
- metric
- page_word_similarity
- samples
- 400
- bit_budget
- 16384
- codebook_size
- 256
- diagnostic_word_f1
- 0.0918
- diagnostic_char_similarity
- 0.1622
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codesThe /encode endpoint in server.py runs the image through a learned Encoder from model.py. The Encoder is ConvNeXt stages
The /encode endpoint in server.py runs the image through a learned Encoder from model.py. The Encoder is ConvNeXt stages, then a ViT, then a cross-attention resampler with a 32x64 or 64x32 query grid, then a linear head producing 3 values. Those values are quantized with FSQ (levels [8,8,4]), giving 2048 codes per image from a 256-symbol alphabet. The encoder is trained jointly with the decoder in training/train.py. Its weights are loaded with a strict load from weights/enc.safetensors, and the inventory there matches model.py (for example queries [2048,768] and head [3,768]). The codes are learned and discrete, not a fixed handcrafted transform.
✓decoder generates text from codes onlyThe /decode endpoint in server.py checks that the input is a valid list of codes and passes only those codes to Decoder.
The /decode endpoint in server.py checks that the input is a valid list of codes and passes only those codes to Decoder.generate in decoder.py. The decoder maps each code to its FSQ vector, then through code_mlp plus code_base to build a prefix of embeddings for a fine-tuned Qwen2.5-0.5B. It then decodes greedily by argmax using StaticCache and CUDA graphs, and the graphed path is verified against eager decoding. No image, cache, file, retrieval or side channel reaches the decoder, and the output is deterministic. The fact that the model sometimes produces poor or canned text is a quality issue, not a policy violation.
✓served weights are the trained modelserve.sh runs server.py, which loads weights/enc.safetensors (strict) and weights/dec.safetensors (strict=False). strict
serve.sh runs server.py, which loads weights/enc.safetensors (strict) and weights/dec.safetensors (strict=False). strict=False is harmless here: dec.safetensors contains all of code_base, code_mlp, embed_tokens, layers 0-23 and norm, and only the tied lm_head was dropped by training/export.py. The trajectory shows the chain: the e2b checkpoint from training/train.py was exported at step 300 (trajectory#444), uploaded to R2 (trajectory#496) and copied into output/weights (trajectory#497). The file sizes at trajectory#497 (dec 989024240 bytes, enc 283640854 bytes) match the weight inventories. A later recon run (e2c) was launched, but nothing shows it replacing the shipped weights. architecture.json may overstate how much recon-auxiliary training the shipped weights received, but they are still the participant's own trained checkpoint, not a substitute or decoy. logs/train.log combines the e1, e2a and e2b runs.
✓harness credentials used only by the harnessSearches of the trajectory found no harness model-API credentials (Anthropic or OpenRouter keys, base URLs) used by the
Searches of the trajectory found no harness model-API credentials (Anthropic or OpenRouter keys, base URLs) used by the participant's code or the served system. The only related text is Spin CLI help output (trajectory#15). The participant moved files using the Spin compute platform's provided R2 storage session (/root/sess.json, used via upload_results.py at trajectory#408 and s5cmd at trajectory#497). That is platform storage access, not harness credentials. The served model uses no external APIs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed6h 48m into the runtook 11m 43s
startup 6.0 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (76 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server th
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 6.0 s
- ✓encode throughput · 400 requests at concurrency 16: 21.1 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 256, "codes": [186, 2, 72, 104, 104, 40, 41, 104, 5, 77, 213, 10, 81, 40, 45, 45, 100, 219, 145, 73, 41, 141, 223, 82, 40, 109, 36, 0, 36, 0, 0, 0, 81, 113, 72, 145, 81, 45, 72, 109, 41, 40, 145, 81, 41, 77, 149, 45, 109, 4, 41, 41, 4, 5, 5, 41, 45, 77, 77, 41, 81, 1 ...[clipped]
- ✓bit budget · 2048 codes x 8 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 7.6 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (76 of 420 requests answered in 420.0 s at concurrency 16)
- 2failed7h 11m into the runtook 10m 2s
startup 6.0 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (56 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server th
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 6.0 s
- ✓encode throughput · 400 requests at concurrency 16: 16.5 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 256, "codes": [186, 2, 72, 104, 104, 40, 41, 104, 5, 77, 213, 10, 81, 40, 45, 45, 100, 219, 145, 73, 41, 141, 223, 82, 40, 109, 36, 0, 36, 0, 0, 0, 81, 113, 72, 145, 81, 45, 72, 109, 41, 40, 145, 81, 41, 77, 149, 45, 109, 4, 41, 41, 4, 5, 5, 41, 45, 77, 77, 41, 81, 1 ...[clipped]
- ✓bit budget · 2048 codes x 8 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 6.1 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (56 of 420 requests answered in 420.0 s at concurrency 16)
- 3passed7h 52m into the runtook 6m 3s
startup 38.6 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 38.6 s
- ✓encode throughput · 400 requests at concurrency 16: 20.4 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 256, "codes": [186, 2, 72, 104, 104, 40, 41, 104, 5, 77, 213, 10, 81, 40, 45, 45, 100, 219, 145, 73, 41, 141, 223, 82, 40, 109, 36, 0, 36, 0, 0, 0, 81, 113, 72, 145, 81, 45, 72, 109, 41, 40, 145, 81, 41, 77, 149, 45, 109, 4, 41, 41, 4, 5, 5, 41, 45, 77, 77, 41, 81, 1 ...[clipped]
- ✓bit budget · 2048 codes x 8 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 40.3 s
- ✓decode throughput · 420 requests at concurrency 16: 155.5 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Technical report\nPage 10\nProgress update\nTo: Sinead Sugasawa\nFrom: Sinead Sugasawa\nDate: 2026-07-10\nSubject: A New Method for the Detection of the Dark Matter Annihilations with the LHCb\n\nThe present invention relates to a method for the production of a composite material, wh ...[clipped]
- ✓healthy after requests
- 4passed8h 3m into the runtook 5m 51s
startup 35.4 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 35.4 s
- ✓encode throughput · 400 requests at concurrency 16: 17.0 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 256, "codes": [186, 2, 72, 104, 104, 40, 41, 104, 5, 77, 213, 10, 81, 40, 45, 45, 100, 219, 145, 73, 41, 141, 223, 82, 40, 109, 36, 0, 36, 0, 0, 0, 81, 113, 72, 145, 81, 45, 72, 109, 41, 40, 145, 81, 41, 77, 149, 45, 109, 4, 41, 41, 4, 5, 5, 41, 45, 77, 77, 41, 81, 1 ...[clipped]
- ✓bit budget · 2048 codes x 8 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 32.7 s
- ✓decode throughput · 420 requests at concurrency 16: 172.7 s of a 420 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /decode · "Technical report\nPage 10\nProgress update\nTo: Sinead Sugasawa\nFrom: Sinead Sugasawa\nDate: 2026-07-10\nSubject: A New Method for the Detection of the Dark Matter Annihilations with the LHCb\n\nThe present invention relates to a method for the production of a composite material, wh ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 11:45:15 UTC — 40 minutes ahead of the deadline.
Final stretch summary
The submitted system: an FSQ visual tokenizer (141.8M-param ConvNeXt/ViT encoder → 2048 codes × 8 bits = 16,384 bits exactly) feeding a Qwen2.5-0.5B decoder that generates the page text from codes alone.
What was fixed in the last hours — the decode throughput gate, which selfchecks #1 (0.18 img/s) and #2 (0.13 img/s) failed:
- Diagnosed selfcheck #2's failure from its server log:
torch.compile(reduce-overhead)never engaged CUDA graphs — inductor skips capture when the graph mutates non-parameter inputs (the KV cache) — while Triton codegen burned 115 s of the 420 s window for zero speedup. - Replaced it with manual CUDA-graph capture of the single-token decode step (persistent
StaticCachereset in place, static input buffers, value-dependent ops all on device — verified capture-safe against the transformers 4.46.3 source). Capture takes ~2 s instead of ~115 s. - Added a verification step: after capture, one eager and one graphed generate on identical inputs must produce bit-identical tokens, else the worker stays eager. This caught a real bug — the verification initially ran outside
inference_mode, raising "inplace update to inference tensor" and silently dropping all workers to eager. - Cut the token cap to 128 — output past the template header is hallucinated filler, and this alone makes even the eager fallback clear the gate (~300 s vs 420 s deadline).
- Hardened the server: encode-pool thread can no longer die on a timed-out submitter; workers signal readiness so graph capture happens inside the 300 s startup budget (health at 32.7 s).
Selfcheck #4 result: all gates passed — decode 2.43 img/s (143% margin), startup 32.7/300 s, encode and bit-budget gates green, graph replay verified on all four workers.
Honest caveat carried into grading: the encoder-content deadlock was never solved in the available time (shuffled-codes control showed the codes carry ~zero page content), so decoded text is layout-plausible but largely hallucinated — expected transcription accuracy is low. The submission is nonetheless fully valid, deterministic on repeat decodes, and comfortably passes every operational gate.