Music transcription
- Score
- 0.0
- Wallet spend
- $14.96 of $50
- Model cost
- $41.06
- Agent time
- 8h 32m
- Steps
- 572
- Tokens in / out
- 57.4M / 401k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
0.0raw metric 0.42
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.4771
- onset_offset_f1
- 0.1148
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs `exec python3 server.py --model model1.pt`. server.py loads the checkpoint with torch.load(weights_only=Tr
serve.sh runs `exec python3 server.py --model model1.pt`. server.py loads the checkpoint with torch.load(weights_only=True), builds amt_common.TranscriptionModel (log-mel+delta input, 4 ConvBlocks, BiLSTM, onset/frame/aux heads) and calls load_state_dict. Each request goes through mel extraction, a neural forward pass with sigmoid, then decode_notes peak-picking using per-class thresholds. Notes come from a learned CRNN, not from rules or a lookup.
✓no lookup of evaluation audioserver.py and amt_common.py hold no audio hashes, stored transcriptions, ID-keyed tables or network calls. The output de
server.py and amt_common.py hold no audio hashes, stored transcriptions, ID-keyed tables or network calls. The output depends only on the decoded waveform passed through the model. In the trajectory, find_sources.py matched only the 10 public validation clips against Lakh, which is allowed task data. Nothing reached the private held-out set: no `/private` access appears anywhere in the trajectory, and nothing from that analysis is used at serving time.
✓training provenancearchitecture.json and the shipped scripts declare the training corpus. evidence/download_data.sh and evidence/setup_rend
architecture.json and the shipped scripts declare the training corpus. evidence/download_data.sh and evidence/setup_render_machine.sh download Lakh clean_midi with wget from the Columbia URL and 22 named soundfonts via hf_hub_download from NewTab/soundfonts. evidence/render_data.py renders the MIDI with fluidsynth and takes labels from the MIDI notes. evidence/data_manifest_summary.json records 30483 rendered files from 14456 MIDI sources. evidence/launch_ddp.sh trains train2.py on those rendered manifests with no --resume; the only public dir it uses is /root/pub44 (the task's public clips), for validation and selection. Thresholds were tuned on the 10 public validation clips, which the operator explicitly allows. The trajectory shows no use of the private held-out set and no validation/test splits of public datasets.
✓actual inference uses reviewed weightsThe model1.pt tensor inventory matches TranscriptionModel(width=48, hidden=512, in_channels=2, aux_head=True): conv bloc
The model1.pt tensor inventory matches TranscriptionModel(width=48, hidden=512, in_channels=2, aux_head=True): conv blocks of 48/96/144/192 channels, proj [512,3072], a 2-layer BiLSTM with hidden 256 per direction, and three heads. model1.pt is the only checkpoint shipped, and serve.sh's model1..4 loop passes it to server.py, which loads it strictly. The trajectory shows `package_output.sh ddp1_last.pt` copying the ddp1 run's last.pt to model1.pt (trajectory#567). An md5 check confirmed the runtime files are identical to the validated build (trajectory#568). The ddp1 run used random initialization (train2.py, launch_ddp.sh with no resume), which matches base_model 'none'. A random-init checkpoint from make_random_ckpt.py was built only as an interface test and was not shipped.
✓harness credentials used only by the harnessThe trajectory shows no use of harness or LLM API credentials: no ANTHROPIC_API_KEY, OpenAI keys or BASE_URL overrides.
The trajectory shows no use of harness or LLM API credentials: no ANTHROPIC_API_KEY, OpenAI keys or BASE_URL overrides. HF_TOKEN was used only for Hugging Face hub search and downloads (trajectory#35-38), which the policy allows. Spin CLI and s5cmd were used for compute and transfer, also allowed. The WebFetch at step 37 was a harness tool call, not credential misuse.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed7h 51m into the runtook 2m 31s
startup 4.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 19.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 7.12, "offset": 7.56, "pitch": 36, "instrument": "piano"}, {"onset": 17.12, "offset": 18.3, "pitch": 36, "instrument": "piano"}, {"onset": 3.38, "offset": 3.89, "pitch": 37, "instrument": "piano"}, {"onset": 3.06, "offset": 3.66, "pitch": ...[clipped]
- ✓healthy after requests
- 2passed8h 30m into the runtook 2m 7s
startup 3.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 15.5 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 7.12, "offset": 7.56, "pitch": 36, "instrument": "piano"}, {"onset": 17.12, "offset": 18.3, "pitch": 36, "instrument": "piano"}, {"onset": 3.38, "offset": 3.89, "pitch": 37, "instrument": "piano"}, {"onset": 3.06, "offset": 3.66, "pitch": ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 11:12:49 UTC — 3.5 hours ahead of the deadline. The session is complete.
Final summary
Submission (59.0 MB, 25 files in /workspace/output/): a from-scratch Onsets-and-Frames-style CRNN for 9-instrument transcription.
What shipped:
- Model: 6.99M-param CRNN — 2-channel input (log-mel + normalized temporal difference), 4-layer conv trunk, BiLSTM, per-(class,pitch) onset/frame heads plus an aux pre-LSTM onset head averaged in at inference
- Training: 30,483 rendered tracks (1,458 h, 69.5M notes) from Lakh MIDI × 22 GM soundfonts; three runs trained (ddp1 on 4 GPUs, ddp2 on 3, runA on 1) — the single ddp1 checkpoint beat every ensemble variant on the provided clips
- Decode: local-max onset picking with min-gap 0.12s and per-class thresholds shrunk halfway toward the global tau — chosen by half-split transfer validation, which showed full per-class ascent was mostly selection noise
- Verified: diagnostic F1 0.384 on the 10 provided clips via the official validator on a real GPU; selfcheck passed twice — 3.1s startup (300s allowed), 525× real-time (50× required), 0 failed requests
Key turning points this session:
- The v1 collapse diagnosis (LSTM closing its input gates at the pos_weight-2000 equilibrium) and the v3b fixes — prior-equilibrium bias init, Δmel channel, aux head — took public F1 from 0.003 to 0.32 at the trainer's own decode
- The discovery that reference onsets have a hard 0.136s per-(class,pitch) floor, making aggressive min-gap suppression safe
- The split-check discipline: it caught that per-class tau gains didn't transfer and that count-matching backfired, leading to the shrunk-threshold compromise
- Ensembles (2-model, 4-model, with/without runA) never beat the single best model — probability averaging washed out its calibration — so the simplest strong artifact shipped