Music transcription
- Score
- 73.8
- Wallet spend
- $9.56 of $10
- Model cost
- $33.86
- Agent time
- 9h 44m
- Steps
- 529
- Tokens in / out
- 52.3M / 423k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 1st of 14
Score
73.8raw metric 0.8827
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.8981
- onset_offset_f1
- 0.6724
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs server.py. server.py builds an ensemble of AMT models defined in amtlib.py: an STFT log-frequency frontend
serve.sh runs server.py. server.py builds an ensemble of AMT models defined in amtlib.py: an STFT log-frequency frontend, a conv stem, RoPE transformer blocks and onset/frame/hidden heads (9 classes x 88 pitches). Each model loads the local weights/r3.safetensors (d=512, 8 layers) and weights/r2c.safetensors (d=640, 10 layers) listed in config.json. Window probabilities are averaged and notes come from a deterministic decoder (decode_onsets with peak picking and per-class thresholds). The code makes no remote calls and uses no off-submission models.
✓no lookup of evaluation audio/transcribe base64-decodes the request audio, mixes it to mono, resamples it, runs windowed neural inference and decodes
/transcribe base64-decodes the request audio, mixes it to mono, resamples it, runs windowed neural inference and decodes notes. No hashing, fingerprinting, filename matching or stored answer tables exist in server.py, amtlib.py or config.json. Batching only queues requests for GPU efficiency.
✓training provenancedata_manifest.json and architecture.json declare the training data. It is audio synthesized with FluidSynth from public
data_manifest.json and architecture.json declare the training data. It is audio synthesized with FluidSynth from public Lakh MIDI (LMD-full; the curl command is declared but no download log was retained, so that source is unverified) using GM SF2 soundfonts fetched by training/code/prep_sf.py through hf_hub_download. The task's public example clips (/task/public, mirrored to /workspace/val) and self-rendered synthetic clips are used only for validation, calibration and selection, plus the optional finetune.py on those public examples, which is allowed. No private set and no validation/test splits of public datasets are used. YourMT3 checkpoints were downloaded and tried early (trajectory#14) but abandoned. The shipped tensors match the custom AMT architecture, not YourMT3's.
✓actual inference uses reviewed weightsconfig.json references both shipped weight files and server.py loads each into AMT(d, layers). The inventories match: r3
config.json references both shipped weight files and server.py loads each into AMT(d, layers). The inventories match: r3 has 512-d blocks with qkv 1536x512, head_hid 1024x512 and stem.proj 512x4704. r2c has 640-d blocks with head_hid 1280x640 and stem.proj 640x4704. No decoy checkpoints are present. base_model 'none' / from-scratch is consistent with train.py random initialization. r2c's --init in training/code/run5.sh uses the participant's own r2p checkpoint, and convert.py exports the EMA weights to fp16 safetensors.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, API_KEY, BASE_URL, 'claude -p' and openrouter matched only spin CLI help text. No ste
Trajectory searches for ANTHROPIC, API_KEY, BASE_URL, 'claude -p' and openrouter matched only spin CLI help text. No step reads, copies or exports harness credentials or starts extra CLI sessions. The shipped code (serve.sh, server.py) makes no API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 30m into the runtook 11sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 31m into the runtook 1m 31s
startup 3.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 7.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.3791, "offset": 3.6791, "pitch": 37, "instrument": "piano"}, {"onset": 3.0616, "offset": 3.3816, "pitch": 38, "instrument": "piano"}, {"onset": 2.7483, "offset": 3.0683, "pitch": 40, "instrument": "piano"}, {"onset": 8.0622, "offset": 8. ...[clipped]
- ✓healthy after requests
- 3passed6h 53m into the runtook 3m 45s
startup 5.3 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.3 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 11.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.3762, "offset": 3.6762, "pitch": 37, "instrument": "piano"}, {"onset": 3.0612, "offset": 3.3812, "pitch": 38, "instrument": "piano"}, {"onset": 2.7469, "offset": 3.0669, "pitch": 40, "instrument": "piano"}, {"onset": 8.0596, "offset": 8. ...[clipped]
- ✓healthy after requests
- 4passed9h 42m into the runtook 1m 12s
startup 5.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 10.7 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.3755, "offset": 3.6755, "pitch": 37, "instrument": "piano"}, {"onset": 3.0609, "offset": 3.3809, "pitch": 38, "instrument": "piano"}, {"onset": 2.7466, "offset": 3.0666, "pitch": 40, "instrument": "piano"}, {"onset": 8.0609, "offset": 8. ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The transcription server is submitted: /workspace/output (184.6 MB) was handed in with frontier submit --confirm. The held-out score is not known yet.
- Model: two from-scratch models, a 32M-parameter one and a 59M-parameter one. Each runs over the clip with 20 s and 30 s sliding windows, and the four probability maps are averaged. Peak picking then uses per-class thresholds, chosen from the 10 example clips plus 160 clips I synthesized from Lakh MIDI.
- Example-clip score: 0.728 macro onset F1 through
validate.pyon the local server. That number is optimistic, because the thresholds were selected on those same 10 clips. On the synthetic clips the same system scores about 0.78. - Selfcheck:
frontier selfcheck(second run) passed. Startup took 5.1 s, and the 271-clip pass took 10.7 s against 162.6 s allowed (about 760x real time). - Weak classes: bass, organ and brass are the weakest on the example clips. For bass and organ, many reference notes get no onset above threshold.
- Provenance: the logs for runs 2 and 4 were lost when their rented machines expired, and
data_manifest.jsonsays so. The larger shipped model was continued from a saved snapshot of run 2. - Not tested: the architecture review did not run in selfcheck, and it could still score the submission zero.