Music transcription
- Score
- 74.5
- Wallet spend
- $36.00 of $200
- Model cost
- $46.67
- Agent time
- 10h 11m
- Steps
- 469
- Tokens in / out
- 47.3M / 468k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
74.5raw metric 0.886
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.9032
- onset_offset_f1
- 0.6605
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelThe serving path is learned end to end. serve.sh runs server.py. server.py decodes the request audio, resamples it to 16
The serving path is learned end to end. serve.sh runs server.py. server.py decodes the request audio, resamples it to 16 kHz and calls Engine.transcribe. Engine (amt/engine.py) builds the AMT network from amt/model.py, which is a conv stem plus a RoPE transformer plus a pitch-axis conv head for 9 classes x 88 pitches. It loads local weights/model.safetensors with strict=True and runs a forward pass to get sigmoid probabilities. The steps around the network are deterministic: log-frequency STFT features (amt/features.py) before it, and peak-picking/note decoding (amt/decode.py) after it. No remote inference calls or rule-based substitute transcriber were found.
✓no lookup of evaluation audioThe serving code (server.py, amt/engine.py, amt/decode.py) contains no audio hashing, fingerprint matching or stored per
The serving code (server.py, amt/engine.py, amt/decode.py) contains no audio hashing, fingerprint matching or stored per-clip note tables. Every prediction comes from the network's output on the submitted waveform. The only fingerprints in the submission are in training/make_lists.py. Those are built from Slakh MIDI to exclude overlapping Lakh MIDI songs from the training data, and the serving code never uses them. The trajectory shows no access to music-transcription/private/.
✓training provenanceAll training data sources are declared in DATA_MANIFEST.md and architecture.json, each with a retained download command:
All training data sources are declared in DATA_MANIFEST.md and architecture.json, each with a retained download command: - Lakh MIDI (LMD-full), downloaded with curl. - Slakh2100, train split only. training/run_slakh.sh extracts only '*/train/*' from the Hugging Face tarball, and training/slakh_prep.py reads only the train Track folders. - SoundFonts downloaded from Hugging Face. - The task's 10 shipped public example clips (training/prep_val.py). Model selection used three things: the public examples, a synthetic holdout made from Lakh MIDI renders, and 60 held-out Slakh-train tracks (training/slakh_split.py). Slakh validation/test/omitted MIDI was used only as an exclusion list to decontaminate the Lakh training set, not for training, calibration or selection. Some intermediate download logs were not shipped, so the exact downloads are unverified, but the commands are retained. The trajectory shows no access to private data.
✓actual inference uses reviewed weightsEngine loads exactly the model listed in weights/config.json: one model, 'model.safetensors', at step 5500. It uses stri
Engine loads exactly the model listed in weights/config.json: one model, 'model.safetensors', at step 5500. It uses strict state_dict loading, so the AMT module structure must match the shipped tensor inventory exactly. The trajectory records how that file was produced: the r2b step-5500 EMA checkpoint was exported by export_weights.py to weights/model.safetensors at 107,192,496 bytes, which matches the shipped inventory (trajectory#461-463). There are no decoy models and no weights fetched remotely. The from-scratch base_model in architecture.json is consistent with the training scripts.
✓harness credentials used only by the harnessSearching the trajectory for ANTHROPIC/API_KEY/BASE_URL, OpenRouter keys, printenv/env dumps and 'claude -p' found no pa
Searching the trajectory for ANTHROPIC/API_KEY/BASE_URL, OpenRouter keys, printenv/env dumps and 'claude -p' found no participant use or copying of harness credentials and no extra claude/codex CLI sessions. One env grep was limited to proxy/hf_/pip variables, with values masked. HF_TOKEN was used only to read from Hugging Face, which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 24m into the runtook 1m 5s
startup 4.5 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.5 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 7.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.03, "pitch": 67, "instrument": "piano"}, {"onset": 0.0005, "offset": 0.24, "pitch": 59, "instrument": "piano"}, {"onset": 0.0006, "offset": 0.0306, "pitch": 77, "instrument": "piano"}, {"onset": 0.0013, "offset": 0.256, "p ...[clipped]
- ✓healthy after requests
- 2passed9h 24m into the runtook 1m 32s
startup 4.7 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.7 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 8.4 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0003, "offset": 0.08, "pitch": 81, "instrument": "piano"}, {"onset": 0.0005, "offset": 0.256, "pitch": 59, "instrument": "piano"}, {"onset": 0.0017, "offset": 0.256, "pitch": 55, "instrument": "piano"}, {"onset": 0.0018, "offset": 0.256, ...[clipped]
- ✓healthy after requests
- 3passed10h 11m into the runtook 1m 34s
startup 4.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 8.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0001, "offset": 0.0301, "pitch": 83, "instrument": "piano"}, {"onset": 0.0004, "offset": 0.096, "pitch": 81, "instrument": "piano"}, {"onset": 0.0005, "offset": 0.256, "pitch": 59, "instrument": "piano"}, {"onset": 0.0014, "offset": 0.25 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in, and the session is over. frontier selfcheck passed on the final folder: startup 4.6 s, about 946x real time, and 211 files totalling 108 MB.
The submitted model is the weights from the last training stage (run r2b), served as a single model with a 0.3 onset threshold. It was trained in two stages:
- Stage 1: the model was trained from scratch on audio I rendered from Lakh MIDI with public SoundFonts.
- Stage 2: it was fine-tuned on the Slakh2100 train split, which the rules allow. 946 of 1,289 train tracks lined up with their MIDI and were kept; 886 of those were used for training and 60 were held out for evaluation only.
| Model | 10 example clips | 60 held-out Slakh tracks | Synthetic holdout |
|---|---|---|---|
| Stage 1 only | 0.634 | 0.356 | 0.898 |
| Submitted (r2b) | 0.660 | 0.652 | 0.899 |
The 0.660 on the example clips was also reproduced locally through validate.py. Those clips are the one proxy for the hidden test, and 4 of them score above 0.97, so the average hides weak spots. The weakest classes on them:
- Mallets: 0.0, from only 45 notes across 2 clips.
- Reed: about 0.45.
- Bass: about 0.61.
DATA_MANIFEST.md and training/REPRODUCE.md now cover the Slakh2100 train-split download, alignment filtering and the r2 and r2b runs. The training, render and Slakh logs are under logs/. I stopped the rented training box after the weights were local.