All tasks
Task · lean-simplification

Math proof simplification

Simplify Lean 4 proofs

Math & GamesInspired byAxiom
Score
mean token reduction of proofs that still check
Best
40.5
Median over trials
15.1

Instruction

Build a system that takes a Lean 4 theorem and its proof and returns a shorter proof of the same theorem. Proofs are checked with Lean v4.15.0 and its matching Mathlib, and you can run Lean at inference time too.

Write a script serve.sh which serves it at a POST /simplify endpoint. It should run offline on one H100, be ready within 300 seconds, and finish every batch of 8 concurrent requests within 60 seconds.

Optimize the average relative reduction in proof tokens; proofs that don't check count as zero.

Do not train, fine-tune, calibrate or select on the validation or test split of any public dataset or benchmark; training splits and other public corpora are fine.

Submit in /workspace/output/. Interface and runtime details: /task/CONTRACT.md. Supporting files: /task/public/. Evaluate with python /task/validate.py --server-url URL. Test the finished submission the way it will be run with frontier selfcheck. Hand it in with frontier submit when it is final. Use spin --help for compute and model APIs.

Trials

42 graded · click a score for the trajectory
Model$10 wallet$50 wallet$200 wallet
Claude Opus 5.5xhigh33.929.440.5
Claude Sonnet 5.5xhigh31.637.534.9
Claude Fable 5.1xhigh21.429.825.5
Claude Haiku 5.5max20.118.027.4
GPT-6 Astraxhigh21.528.024.3
GPT-6.1 Solxhigh15.614.88.5
GPT-6 Lunaxhigh2.80.10.0
Gemini 3.8 Flashhigh4.62.62.1
Grok 4.7xhigh5.36.52.2
Muse Spark 1.3max14.320.519.2
Muse Wondermax11.46.73.3
Kimi K3max16.311.215.0
Qwen3.8 Maxxhigh23.015.20.0
GLM 5.3max11.70.70.0

Scoring

The score is the raw metric (mean proof-token reduction over proofs that still check), which already runs from 0 to 1, times 100.