OCR latent compression
- Score
- 3.3
- Wallet spend
- $5.40 of $50
- Model cost
- $20.63
- Agent time
- 9h 8m
- Steps
- 272
- Tokens in / out
- 62.4M / 219k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 11th of 14
Score
3.3raw metric 0.02959
The raw metric (word-level edit similarity of the decoded text) is rescaled so that empty text scores 0 and unmodified DeepSeek-OCR reading the page image (0.893) scores 100, clamped to that range.
Verifier details
- metric
- page_word_similarity
- samples
- 400
- bit_budget
- 16384
- codebook_size
- 16
- diagnostic_word_f1
- 0.0593
- diagnostic_char_similarity
- 0.1964
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned discrete visual codesThis check passes. In server.py, /encode runs preprocess_image and then extract_visual_features, using DeepSeek-OCR's SA
This check passes. In server.py, /encode runs preprocess_image and then extract_visual_features, using DeepSeek-OCR's SAM and CLIP encoders plus the projector. The features then go through the learned Compressor in common.py: LayerNorm and positional embedding, adaptive pooling to 1024 slots, an MLP, a 2-layer TransformerEncoder and a Linear to_fsq layer. fsq_quantize turns each slot into 4 values with 16 levels, giving 4096 codes in the range 0-15 (16384 bits). That matches codebook_size=16 in architecture.json. The compressor weights (compressor.*) are in the tokenizer.pt inventory and were trained by train.py; train.log shows a cross-entropy loss, a reconstruction loss and an FSQ penalty. The encode path does no OCR and produces no text.
✓decoder generates text from codes onlyThis check passes. In server.py, decode_batch pads or truncates the request's codes to 4096 and clamps them to 0-15. The
This check passes. In server.py, decode_batch pads or truncates the request's codes to 4096 and clamps them to 0-15. The Decompressor in common.py maps the codes back to slot embeddings. generate_from_slots then builds its input from only the BOS token, those slot embeddings and the fixed 'Free OCR.' prompt, decodes greedily with a no-repeat-ngram ban, and returns tokenizer.decode output. I found no shared file, cache, transcript store, retrieval step or image access between /encode and /decode. The text depends only on the submitted codes.
✓served weights are the trained modelThis check passes. serve.sh sets MODEL_DIR=weights and CKPT=tokenizer.pt. In server.py, load_all loads the declared Deep
This check passes. serve.sh sets MODEL_DIR=weights and CKPT=tokenizer.pt. In server.py, load_all loads the declared DeepSeek-OCR base from weights/; that safetensors inventory matches the base architecture. It then applies LoRA r=8, calls train.load_checkpoint(tokenizer.pt), merges the LoRA weights and enables the grouped-MoE fast path. The trajectory confirms which checkpoint was shipped. Step 240 trained v9 starting from ckpt_v8. Step 243 placed v8 in output, but step 252 switched the upload script to /workspace/ckpt_v9/tokenizer.pt, step 256 re-ran that upload, and step 257 copied tokenizer_v9.pt (294M) to output/tokenizer.pt. That overwrote v8. The 294M size matches the 307.5MB inventory, which contains compressor tensors, LoRA tensors and others.model.norm.weight, consistent with the stage-2 checkpoint format in train.py. Step 258 appended train_v9.log to train.log and updated training_config.json to describe the stage-1 plus stage-2 lineage. The training lineage is consistent.
✓harness credentials used only by the harnessThis check passes. The trajectory contains no search hits for Anthropic or OpenAI keys, no use of OpenRouter inference,
This check passes. The trajectory contains no search hits for Anthropic or OpenAI keys, no use of OpenRouter inference, and no dumping of environment variables. The participant used only participant-facing tools: the HF_TOKEN that the task prompt says was provided, and temporary S3 credentials from 'spin transfer credentials' (steps 241 and 256) to move checkpoints. No harness-only credential was used, and the shipped code contains no API keys or remote calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed8h 25m into the runtook 12m 9s
startup 13.6 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (104 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 13.6 s
- ✓encode throughput · 400 requests at concurrency 16: 68.6 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16, "codes": [15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 14, 15, 15, 15, 14, 15, 15, 15, 1 ...[clipped]
- ✓bit budget · 4096 codes x 4 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 13.2 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (104 of 420 requests answered in 420.0 s at concurrency 16)
- 2failed8h 41m into the runtook 11m 23s
startup 13.7 s of 300 s; 1/2 gates passed; failed decode throughput: Batch exceeded 420s deadline; score is zero (352 of 420 requests answered in 420.0 s at concurrency 16); The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server
9 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 13.7 s
- ✓encode throughput · 400 requests at concurrency 16: 62.1 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16, "codes": [15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 14, 15, 15, 15, 14, 15, 15, 15, 1 ...[clipped]
- ✓bit budget · 4096 codes x 4 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 11.1 s
- ✗decode throughput · Batch exceeded 420s deadline; score is zero (352 of 420 requests answered in 420.0 s at concurrency 16)
- 3passed8h 57m into the runtook 10m 17s
startup 13.8 s of 300 s; 2/2 gates passed; The encode batch sends each of the 100 public images about 4 times and the decode batch their code lists, so a server that caches repeated images or code lists measures faster here than on grading's distinct held-out images.
11 gates
- ✓review prerequisites
- ✓architecture.json
- ✓startup · GET /health answered 200 after 13.8 s
- ✓encode throughput · 400 requests at concurrency 16: 61.8 s of a 400 s deadline, timed from a fresh instance's readiness as in grading
- ✓POST /encode · {"codebook_size": 16, "codes": [15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 14, 15, 15, 15, 14, 15, 15, 15, 1 ...[clipped]
- ✓bit budget · 4096 codes x 4 bits for one public image; every image must fit
- ✓healthy after requests
- ✓startup · GET /health answered 200 after 13.6 s
- ✓decode throughput · 420 requests at concurrency 16: 301.8 s of a 420 s deadline, timed from a fresh instance's readiness as in grading; 80 code lists decoded to different texts when sent more than once (decoding must be deterministic; grading keeps a repeated code list's lower score)
- ✓POST /decode · "Under review\nPage 7\nThe song was written by\nThe song was written by\nThe song was written by\nThe song is a cover of the song \" I 'm Not the Only One \" by the band The Beatles . The song was written by the band 's lead singer Paul McCartney and was recorded by the band in 1970 . ...[clipped]
- ✓healthy after requests