OCR latent compression
- Score
- 7.1
- Wallet spend
- $9.98 of $10
- Model cost
- $80.00
- Agent time
- 10h 49m
- Steps
- 702
- Tokens in / out
- 238.2M / 330k
- Self-checks
- 0 passed of 4 used
- Rank at $10 wallet
- 9th of 14
Score
7.1raw metric 0.06336
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
Verifier details
- metric
- page_word_similarity
- samples
- 400
- bit_budget
- 16384
- codebook_size
- 16777216
- diagnostic_word_f1
- 0.1580
- diagnostic_char_similarity
- 0.2506
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codesOn the /encode path in server.py, the PNG is decoded and resized to a fixed canvas by Q.preprocess. It then goes through
On the /encode path in server.py, the PNG is decoded and resized to a fixed canvas by Q.preprocess. It then goes through the DeepSeek-OCR SAM+CLIP vision encoder and projector (QuantOCR.vision_tokens in qmodel.py), then the trained FSQ quantizer: Linear(1280->8), tanh, then rounding to 8 levels per dimension. The result is mixed-radix integer codes with an alphabet of 2^24 = 16,777,216, which matches codebook_size in architecture.json. There is no OCR or text step anywhere on this path. The shipped weights contain the trained quantizer.in_proj and quantizer.out_proj tensors. Code counts are 660 for documents and 640 for screenshots, so the worst case is 660x24 = 15,840 bits, under 16,384. Trajectory step 638 shows the server test returning 660 codes and 15,840 bits.
✓decoder generates text from codes onlyOn the /decode path in server.py, the request supplies only the integer codes. The grid shape is inferred from the numbe
On the /decode path in server.py, the request supplies only the integer codes. The grid shape is inferred from the number of codes. The codes go through quantizer.from_codes and the out_proj MLP, newline and separator embeddings are added, and a fixed '\nFree OCR.' prompt is appended. Then FastDecoder in decode_fast.py runs greedy argmax over the merged DeepseekV2 LLM, and tokenizer.decode produces the text. Batching groups requests by grid and keeps rows independent. The static KV cache is overwritten at each prefill. There is no persistent cache, file, database or side channel that could carry per-image text from encode to decode.
✓served weights are the trained modelserve.sh runs server.py. server.py calls qmodel.load_merged on weights/, which strictly loads weights/model.safetensors,
serve.sh runs server.py. server.py calls qmodel.load_merged on weights/, which strictly loads weights/model.safetensors, including the quantizer.* keys. The weight inventory (all BF16, standard DeepSeek-OCR keys plus 6 quantizer tensors) matches the module definitions. The provenance chain in the trajectory is: - Stage B LoRA training run (trajectory#327-329). - Merged export of the step-1000 checkpoint via the shipped train.save_merged (trajectory#323, #330). - md5 of the output model.safetensors matches merged1000 (trajectory#638). - The output file has 2717 keys including the quantizer (trajectory#677). The base model is declared with a pinned revision in architecture.json and STARTING_ARTIFACT.json. TRAINING_NOTES.md has minor inconsistencies: it claims fp32 for the projector and FSQ, and the 1500-step run was stopped at step 1000. These are documentation issues, not violations.
✓harness credentials used only by the harnessI searched the trajectory for API keys, base URLs, openrouter, the kimi CLI, openai imports, printenv and curl usage. No
I searched the trajectory for API keys, base URLs, openrouter, the kimi CLI, openai imports, printenv and curl usage. None of it shows the participant using the harness's model credentials. The only network checks were a HEAD request to huggingface.co and a check for HF_TOKEN. Compute was reached over ssh with Spin-provided keys. No shipped code references harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed5h 51m into the runtook 1m 9sfree (stopped before any GPU time)
failed snapshot: Submission must contain serve.sh
1 gates
- ✗snapshot · Submission must contain serve.sh
- 2failed6h 32m into the runtook 1m 27sfree (stopped before any GPU time)
failed architecture.json: Evidence path is not a regular submission file: assets_train/train_B.log
2 gates
- ✓review prerequisites
- ✗architecture.json · Evidence path is not a regular submission file: assets_train/train_B.log
- 3failed7h 46m into the runtook 8m 8s
failed startup: Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status": "loading"}'); end of its log: Exception in thread Thread-1 (load): Traceback (most recent call last): File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner self.run() Fi
3 gates
- ✓review prerequisites
- ✓architecture.json
- ✗startup · Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status": "loading"}'); end of its log: Exception in thread Thread-1 (load): Traceback (most recent call last): File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner self.run() File "/usr/lib/python3.12/threading.py", line 1010, in run serving on 0.0.0.0:8000 self._target(*self._args, **self._kwargs) File "/submission/server.py", line 61, in load model = Q.load_merged(MODEL_DIR, device=DEVICE) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/qmodel.py", line 128, in load_merged base = load_base_model(model_dir, dtype=dtype, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/qmodel.py", line 97, in load_base_model with open(os.path.join(weights_dir, "config.json")) as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/submission/weights/config.json'
- 4failed7h 56m into the runtook 12m 38s
startup 38.7 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (76 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server t
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 38.7 s
- ✓encode throughput · 400 requests at concurrency 16: 39.9 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16777216, "codes": [13788308, 9577636, 14295348, 9577636, 9577636, 9577636, 9577636, 13771938, 13772084, 13772068, 14296246, 13772068, 13771044, 9577764, 9635108, 10093876, 14360738, 14360738, 9577764, 13837604, 13836726, 9511222, 9576756, 13697332, 13771062, 9520404 ...[clipped]
- ✓bit budget · 660 codes x 24 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 43.9 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (76 of 420 requests answered in 420.0 s at concurrency 16)
- 5failed8h 49m into the runtook 10m 10s
failed startup: Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status": "loading"}'); end of its log: d step failed, falling back to eager: The expanded size of the tensor (1) must match the existing size (10) at non-singleton dimension 0. Target sizes: [1, 10, 128]
3 gates
- ✓review prerequisites
- ✓architecture.json
- ✗startup · Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status": "loading"}'); end of its log: d step failed, falling back to eager: The expanded size of the tensor (1) must match the existing size (10) at non-singleton dimension 0. Target sizes: [1, 10, 128]. Tensor sizes: [10, 1, 128] Exception in thread Thread-1 (load): Traceback (most recent call last): File "/submission/server.py", line 83, in load dec.generate(emb, max_new_tokens=8) File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/decode_fast.py", line 188, in generate logits = self._step_fn(emb, pos_t, mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/decode_fast.py", line 121, in _step_eager self.kc[i, :B, :, pos_t] = k.squeeze(1) ~~~~~~~^^^^^^^^^^^^^^^^^ RuntimeError: The expanded size of the tensor (1) must match the existing size (10) at non-singleton dimension 0. Target sizes: [1, 10, 128]. Tensor sizes: [10, 1, 128] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner self.run() File "/usr/lib/python3.12/threading.py", line 1010, in run self._target(*self._args, **self._kwargs) File "/submission/server.py", line 88, in load dec.generate(emb, max_new_tokens=8) File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/submission/decode_fast.py", line 188, in generate logits = self._step_fn(emb, pos_t, mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/decode_fast.py", line 121, in _step_eager self.kc[i, :B, :, pos_t] = k.squeeze(1) ~~~~~~~^^^^^^^^^^^^^^^^^ RuntimeError: The expanded size of the tensor ...[clipped]
- 6failed9h 43m into the runtook 12m 50s
startup 47.1 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (59 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server t
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 47.1 s
- ✓encode throughput · 400 requests at concurrency 16: 43.8 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16777216, "codes": [13788308, 9577636, 14295348, 9577636, 9577636, 9577636, 9577636, 13771938, 13772084, 13772068, 14296246, 13772068, 13771044, 9577764, 9635108, 10093876, 14360738, 14360738, 9577764, 13837604, 13836726, 9511222, 9576756, 13697332, 13771062, 9520404 ...[clipped]
- ✓bit budget · 660 codes x 24 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 38.9 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (59 of 420 requests answered in 420.0 s at concurrency 16)
The agent's last message before its session ended.
No server-side errors — just slow CPU warmup. Components are all verified. Confirming submission: