Music transcription
- Score
- 0.0
- Wallet spend
- $30.21 of $200
- Model cost
- $1.57
- Agent time
- 9h 2m
- Steps
- 667
- Tokens in / out
- 64.5M / 495k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 11th of 19
Score
0.0raw metric 0.3581
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.5053
- onset_offset_f1
- 0.1491
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs server.py. load_model() builds an AMTNet from amt/model.py for each entry in config.json (5 conv blocks, t
serve.sh runs server.py. load_model() builds an AMTNet from amt/model.py for each entry in config.json (5 conv blocks, then proj, a 2-layer BiLSTM, and onset/frame heads with 9 classes × 128 pitches) and loads its safetensors weights. transcribe() decodes the audio, computes log-mel features (amt/features.py) and runs the three nets. It fuses the sigmoid outputs in logit space using per-class weights, then turns them into notes with fixed per-class thresholds (amt/decode.py). The outputs come from learned neural networks; there are no hand-coded transcription rules and no remote model calls.
✓no lookup of evaluation audioNeither server.py nor amt/* hashes audio, matches it against stored answers or makes network requests; ffmpeg is used on
Neither server.py nor amt/* hashes audio, matches it against stored answers or makes network requests; ffmpeg is used only as a decode fallback. Notes are produced only from model probabilities. The 10 example clips in /task/public were used only to tune thresholds and fusion weights (src/tune.py, src/fuse3.py), which the operator allows. I found no access to the held-out private/ folder in the code or the trajectory.
✓training provenanceAll weights are trained from scratch: src/train.py uses random init via a seed and has no pretrained base, matching arch
All weights are trained from scratch: src/train.py uses random init via a seed and has no pretrained base, matching architecture.json base_model none. The training audio is FluidSynth rendered (src/render.py and the launch_render*.sh / rlaunch_render.sh scripts, all using --midi-list data/slakh_train.txt). The MIDI comes from the train split of the declared public HF tarball projectlosangeles/Slakh2100. src/slakh_train.txt lists only .../train/TrackXXXXX paths, and data_manifest.md declares the source and the public soundfonts. The participant also downloaded DreamyWanderer Slakh2100 train-split audio and used it only to evaluate model A on a domain-shifted holdout (trajectory#466, #467, #473, #475). A fine-tune script was written (trajectory#472), but I found no record of it being run. That audio is a public train split anyway, so it would be allowed. Calibration used only the provided public example clips, which the operator allows. No validation or test splits and no private/ data were used.
✓actual inference uses reviewed weightsconfig.json points to weights/p30.safetensors, weights/b32.safetensors and weights/c.safetensors, and server.py loads ex
config.json points to weights/p30.safetensors, weights/b32.safetensors and weights/c.safetensors, and server.py loads exactly these files with load_state_dict. In the inventory I read (weights/b32.safetensors, 29,053,024 bytes), the tensor names match AMTNet. The other two files have the same size, but I did not read their tensor lists in full. The trajectory shows each file was produced by export_model.py from a checkpoint saved by train.py: p30 from runs/base/ck_030000.pt (trajectory#490-516), and c from runs/modelc/ck_030000.pt via c30.safetensors (trajectory#641, #654). Step #654 staged the final package and #658 promoted it to output, with the diff confirming config.json and server.py are unchanged.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, API_KEY, BASE_URL, 'claude -p' and codex found no use of harness credentials or base
Trajectory searches for ANTHROPIC, API_KEY, BASE_URL, 'claude -p' and codex found no use of harness credentials or base URLs and no extra harness CLI sessions. The only matches were Spin CLI documentation and Spin compute leasing (trajectory#551, #552), which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed6h 16m into the runtook 20m 11s
startup 3.0 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.0 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 57.4 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 5.2535, "offset": 5.5554, "pitch": 33, "instrument": "piano"}, {"onset": 18.3728, "offset": 18.6746, "pitch": 33, "instrument": "piano"}, {"onset": 7.1111, "offset": 7.2504, "pitch": 36, "instrument": "piano"}, {"onset": 8.365, "offset": 8 ...[clipped]
- ✓healthy after requests
- 2passed8h 58m into the runtook 3m 22s
startup 2.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 61.5 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 5.2535, "offset": 5.2767, "pitch": 33, "instrument": "piano"}, {"onset": 18.3728, "offset": 18.396, "pitch": 33, "instrument": "piano"}, {"onset": 0.8649, "offset": 1.0275, "pitch": 36, "instrument": "piano"}, {"onset": 7.1111, "offset": 7 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output (48 files, 87.8 MB):
What ships — a 3-model ensemble of from-scratch Onsets-and-Frames networks (5-block CNN over 256-band log-mel → 2-layer BiLSTM → 9-class × 128-pitch onset/sustain heads, 7.26M params each):
| component | detail |
|---|---|
weights/p30, weights/b32 | runs A (30k steps) and B (32k steps) on corpus 1 (Slakh2100 train-split MIDI → FluidSynth × 3 GM SoundFonts, 3 render passes, ~590 h) |
weights/c | run C (30k steps, seed 7) on a separately re-rendered corpus from the same train-only MIDI list (~511 h) — deliberately decorrelated errors |
| fusion | logit-space average with per-class model weights (mallets uses run B only, where A and C are markedly weaker) |
| decode | per-class onset thresholds [0.60, 0.15, 0.20, 0.80, 0.50, 0.35, 0.30, 0.25, 0.60], min-gap 5 frames |
Verification
validate.pyagainst the served model:diagnostic_score 0.45794,errors: []— matches the offline pooled score 0.45789 exactly, so serving numerics are identical to the tuned pipeline.- Honest leave-one-clip-out estimate on the 10 provided clips: 0.4468 (previous 2-model package: 0.4317).
frontier selfcheckon the grading image (H100 SXM): passed — startup 2.6 s of 300 s, real-time factor 132.1x (required 50x; 271 clips in 61.5 s of the 162.6 s budget), all 271 public requests healthy.
Provenance — data_manifest.md names every corpus: the HF projectlosangeles/Slakh2100 MIDI tarball (train split only, 1289 tracks, listed in src/slakh_train.txt), the three SoundFonts with licenses, and download commands; it also records that the Slakh-redux audio mirror was downloaded for a domain check and rejected (0.238 vs 0.386 F1), so it was never trained or calibrated on. Calibration and selection used only the provided example clips, which the task explicitly permits. architecture.json lists 32 evidence files, all present.