Music transcription
- Score
- 61.0
- Wallet spend
- $43.29 of $50
- Model cost
- $52.64
- Agent time
- 8h 4m
- Steps
- 501
- Tokens in / out
- 193.5M / 318k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 3rd of 14
Score
61.0raw metric 0.8256
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.8683
- onset_offset_f1
- 0.7614
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs server.py offline. server.py builds Transcriber (engine.py), which constructs the YourMT3 Perceiver-TF MoE
serve.sh runs server.py offline. server.py builds Transcriber (engine.py), which constructs the YourMT3 Perceiver-TF MoE encoder and T5 decoder modules from ymt3_cfg.json and ymt3src. It loads the state dict from weights/model.pt and fails on missing keys. Transcription is greedy autoregressive decoding followed by detokenization. The weight inventory for weights/model.pt is a fully inspected PyTorch state dict of about 188.5 MB with the expected pre_encoder, encoder and decoder tensors. This is a learned neural transcription model, not rules or a lookup.
✓no lookup of evaluation audioserver.py decodes the base64 audio, downmixes it to mono, resamples it to 16 kHz and passes the batch straight to model.
server.py decodes the base64 audio, downmixes it to mono, resamples it to 16 kHz and passes the batch straight to model.transcribe. There is no hashing, no remote call, no reference to stored answers and no access to music-transcription/private/. The trajectory shows no access to the held-out set: the only 'private' matches are SSH keys for the Spin GPU boxes.
✓training provenanceThe base checkpoint is the public mimbres/YourMT3 HF Space last.ckpt, declared in architecture.json and downloaded in th
The base checkpoint is the public mimbres/YourMT3 HF Space last.ckpt, declared in architecture.json and downloaded in the logged commands. The training audio is FluidSynth renders of Slakh2100-redux train-split MIDI only, using public GM soundfonts. train/render.py renders from --midi-dir, and every logged render command points to slakh2100_flac_redux/train (train/logs/commands.md and the trajectory render steps). Model selection used the task's public example clips (/task/public/validation.jsonl), which policy allows, plus synthetic renders from the train split. No public validation or test split, and no private data, was used. The render logs for r9–r14 were not retained, but their commands and sources are declared, so those runs are recorded as declared but unverified.
✓actual inference uses reviewed weightsengine.py loads weights/model.pt with a missing-keys check, and the fast decoder reuses decoder, embedding and lm_head t
engine.py loads weights/model.pt with a missing-keys check, and the fast decoder reuses decoder, embedding and lm_head tensors from the same state dict. The trajectory shows the candidate cand_swa5c.pt was produced by soup.py, averaging the ft5 checkpoints plus final.pt (trajectory#489, trajectory#490). It was copied into output/weights/model.pt at trajectory#492, with a matching md5 at trajectory#496. This matches architecture.json and train/logs/model_selection.md. The serving path loads no other weights.
✓harness credentials used only by the harnessSearches of the trajectory for harness API credentials (ANTHROPIC, API_KEY, BASE_URL, claude invocations) found no use b
Searches of the trajectory for harness API credentials (ANTHROPIC, API_KEY, BASE_URL, claude invocations) found no use by the participant's code. HF_TOKEN was used only for read downloads (trajectory#67), and Spin CLI was used for GPU boxes; policy allows both. The shipped server runs offline and makes no credentialed calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 5m into the runtook 3m 16s
startup 15.3 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 15.3 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 35.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.3679, "offset": 3.6779, "pitch": 37, "instrument": "piano"}, {"onset": 3.0579, "offset": 3.3679, "pitch": 38, "instrument": "piano"}, {"onset": 2.7479, "offset": 3.0579, "pitch": 40, "instrument": "piano"}, {"onset": 8.0538, "offset": 8. ...[clipped]
- ✓healthy after requests
- 2passed3h 37m into the runtook 8m 20s
startup 18.4 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 18.4 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 38.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.3679, "offset": 3.6879, "pitch": 37, "instrument": "piano"}, {"onset": 3.0579, "offset": 3.3679, "pitch": 38, "instrument": "piano"}, {"onset": 2.7479, "offset": 3.0579, "pitch": 40, "instrument": "piano"}, {"onset": 8.0538, "offset": 8. ...[clipped]
- ✓healthy after requests
- 3passed8h 2m into the runtook 2m 8s
startup 18.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 18.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 56.3 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.3679, "offset": 3.6879, "pitch": 37, "instrument": "piano"}, {"onset": 3.0579, "offset": 3.3679, "pitch": 38, "instrument": "piano"}, {"onset": 2.7479, "offset": 3.0579, "pitch": 40, "instrument": "piano"}, {"onset": 8.0538, "offset": 8. ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The only change since selfcheck #3 is this metadata text, which doesn't affect serving. Submitting: