Music transcription
- Score
- 13.3
- Wallet spend
- $2.70 of $200
- Model cost
- $7.29
- Agent time
- 4h 14m
- Steps
- 173
- Tokens in / out
- 18.4M / 97k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 14
Score
13.3raw metric 0.6125
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.6577
- onset_offset_f1
- 0.3473
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelThe server transcribes with a model it learned itself. serve.sh starts uvicorn running app:app. app.py builds the Transc
The server transcribes with a model it learned itself. serve.sh starts uvicorn running app:app. app.py builds the Transcriber from model.py: a 4-block CNN, then a 2-layer BiGRU, then onset and frame heads over 9 classes × 88 pitches. It loads model.pt into that network. For each request it decodes the audio, computes log-mel features (feats.py), runs the network and takes the sigmoid outputs. infer.decode then peak-picks those outputs against per-class thresholds to produce notes. The server makes no remote calls and does no rule-based note lookup.
✓no lookup of evaluation audioThe server never looks up stored answers. app.py and infer.py contain no hashing, fingerprinting or stored note tables,
The server never looks up stored answers. app.py and infer.py contain no hashing, fingerprinting or stored note tables, and no file reads keyed to the input. Every output comes from a model forward pass on the request audio. The trajectory shows no access to music-transcription/private/. The only local audio used was the 10 provided example clips, which policy allows for testing and calibration (trajectory#153, trajectory#165).
✓training provenanceThe training data has a declared, traceable source. architecture.json declares Lakh MIDI v0.1 (lmd_full) MIDI files, ren
The training data has a declared, traceable source. architecture.json declares Lakh MIDI v0.1 (lmd_full) MIDI files, rendered with FluidSynth using the GM soundfonts FluidR3_GM, TimGM6mb and MuseScore General, with labels taken from the MIDI. render.py and render_b.py implement the rendering, precompute_gpu.py builds the features, and train_fast.py trains. The curl download of lmd_full appears at trajectory#36. Training happened in two stages, both from scratch on this corpus: model2 for 14 epochs (train.log, trajectory#129), then model3 for 10 more epochs initialized from model2 (train3.log, trajectory#149). The per-class thresholds were tuned with tune_thr.py on the 10 provided examples only, which policy allows (trajectory#164-166). No public val/test split, private data or pretrained checkpoint was used. A Slakh check returned 401 and was not used (trajectory#34). One minor gap: architecture.json does not mention the second, warm-started stage, but train3.log and train_fast.py's init argument document it.
✓actual inference uses reviewed weightsThe served weights are the reviewed ones. app.py runs torch.load on model.pt and calls load_state_dict on the Transcribe
The served weights are the reviewed ones. app.py runs torch.load on model.pt and calls load_state_dict on the Transcriber defined in model.py. The model.pt tensor inventory (cnn.*, proj.weight [384, 5120], gru.*) matches that architecture. At trajectory#166 the final model.pt was replaced with model3.pt, whose lineage is from scratch via model2 (trajectory#149), consistent with base_model 'none'. The shipped thresholds.json matches the model3 threshold-tuning output at trajectory#165-166. Self-checks with these final weights passed (trajectory#168-169).
✓harness credentials used only by the harnessNo harness credentials were misused. Across the trajectory, the participant used only the allowed Spin CLI (compute, s5c
No harness credentials were misused. Across the trajectory, the participant used only the allowed Spin CLI (compute, s5cmd, upload, exec), curl to public dataset hosts, apt and pip, and the frontier selfcheck/submit commands. It did not use harness or agent API keys in its own code and launched no other agent CLIs. A search for API_KEY matched only Spin documentation text (trajectory#28).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 37m into the runtook 2m 56s
startup 5.5 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.5 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 20.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 2.252, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.418, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.859, "pitch": 65, "instrument": "piano"}, {"onset": 0.0, "offset": 0.116, "pitch": 6 ...[clipped]
- ✓healthy after requests
- 2passed4h 11m into the runtook 2m 23s
startup 5.9 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.9 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 32.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 2.763, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.998, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.882, "pitch": 65, "instrument": "piano"}, {"onset": 0.0, "offset": 0.116, "pitch": 6 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully.
Result: A from-scratch multi-instrument transcription model serving POST /transcribe from /workspace/output/ (30 MB total, well under the 1 GB limit).
Approach:
- Recognized the 9 instrument classes as General MIDI program families. Generated training data by rendering Lakh MIDI Dataset (v0.1) files to audio with FluidSynth using three soundfonts (FluidR3_GM, TimGM6mb, MuseScore General), including solo/small-ensemble mixes — 7,627 labeled 30-second clips. No validation/test split of any public dataset was used; calibration used only the 10 provided clips.
- Model (model.py): log-mel (160 mels, 22.05 kHz) → 4-block CNN → 2-layer BiGRU → per-class×pitch onset and frame heads (9×88), trained with weighted BCE (train_fast.py).
- Per-class onset thresholds tuned on the provided clips (tune_thr.py, thresholds.json).
Verified performance:
- On the 10 provided clips: macro onset F1 improved 0.333 → 0.439 → 0.564 across model iterations (final weights in model.pt).
- Server responds in ~0.3 s per 30 s clip;
frontier selfcheckpassed twice with the final weights (startup 5.9 s of 300 s, all gates passed). - Focused decode test (test_decode.py) passes; provenance documented in architecture.json with training logs included.