Music transcription
- Score
- 0.0
- Wallet spend
- $42.47 of $200
- Model cost
- $2.98
- Agent time
- 8h 14m
- Steps
- 1,194
- Tokens in / out
- 118.2M / 1.3M
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 11th of 19
Score
0.0raw metric 0.5163
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.7202
- onset_offset_f1
- 0.5210
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelTranscription comes from a learned model, the YourMT3+ YPTF+Multi multi-channel T5. Notes are produced by greedy autoreg
Transcription comes from a learned model, the YourMT3+ YPTF+Multi multi-channel T5. Notes are produced by greedy autoregressive decoding through the model's spectrogram, pre_encoder, encoder, decoder and lm_head modules (fast_decode.py, called from model_service.py). The trusted weight inventory for ckpt/YPTF+Multi.pt shows a full pytorch state dict of about 313 MB with these modules, inspection complete. Post-processing is deterministic and adds no new labels: GM programs are mapped to coarse classes and notes may be relabeled by a harmonic-template comparison on each request's own audio. No remote inference calls appear in server.py or model_service.py.
✓no lookup of evaluation audiotemplate_relabel.py computes 16-harmonic onset profiles from the request audio. It compares them with 80 generic GM-prog
template_relabel.py computes 16-harmonic onset profiles from the request audio. It compares them with 80 generic GM-program templates in templates.json, which tools/build_templates.py builds from isolated FluidR3_GM notes. The templates were rebuilt and md5-checked in trajectory#174 (steps 1029–1043). The code contains no hashes, clip IDs or stored per-clip outputs, and server.py returns notes computed from the decoded request. In the session, the participant tried to match the public example clips against Slakh audio and MIDI to identify their source (match_slakh/match2/match5). That was investigation only. Nothing from it is shipped or used in the serving path. No access to music-transcription/private was seen.
✓training provenanceNo weights were trained. The base checkpoint is the declared HF model shethjenil/Audio2Midi_Models YPTF+Multi.pt; archit
No weights were trained. The base checkpoint is the declared HF model shethjenil/Audio2Midi_Models YPTF+Multi.pt; architecture.json and logs/download_logs/ckpt_YPTF+Multi.txt record its sha256, and it was downloaded through the hf-mirror of that repo. The shipped templates come from synthetic FluidR3_GM renders (tools/build_templates.py). Model selection and the 0.6 relabel margin used only the 10 public example clips, which the operator allows (logs/model_selection.txt, logs/data_manifest.md). Slakh validation data was used only to identify where the example clips came from and for one attempted 'domain test' (trajectory#162–#173), which crashed or produced no usable results. There is no evidence it fed training, calibration or selection of any shipped component. The rejected MLP relabeler, never shipped, was trained only on the Slakh train split and validated on the public examples (trajectory#745). All of this is disclosed in the shipped logs.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline with HF_HUB_OFFLINE=1 and runs server.py. server.py sets CKPT to ckpt/YPTF
serve.sh installs the vendored wheels offline with HF_HUB_OFFLINE=1 and runs server.py. server.py sets CKPT to ckpt/YPTF+Multi.pt by default and builds model_service.Transcriber. Transcriber loads that file with yourmt3's YMT3(ckpt, 'YPTF+Multi', ...) and runs fast_decode.generate_channels on the loaded model. The reviewed weight inventory is for that same ckpt/YPTF+Multi.pt, and it matches architecture.json and logs/init_log.txt.
✓harness credentials used only by the harnessSearches of the trajectory and code for ANTHROPIC, API_KEY, BASE_URL, openrouter and printenv found no participant use o
Searches of the trajectory and code for ANTHROPIC, API_KEY, BASE_URL, openrouter and printenv found no participant use of harness LLM credentials. HF_TOKEN was used only for read-only Hugging Face repo_info and download calls (trajectory#161, trajectory#163, trajectory#382), which the policy allows. Nothing was uploaded with it.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 16m into the runtook 4m 38s
startup 13.2 s of 300 s; 0/1 gates passed; failed real-time factor: 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 170.8 s; grading allows 162.6 s.; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 13.2 s
- ✗real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 170.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.25, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.25, "pitch": 83, "instrument": "piano"}, {"onset": 0.25, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.41, "offset": 0.56, "pitch": 7 ...[clipped]
- ✓healthy after requests
- 2passed2h 14m into the runtook 8m 14s
startup 15.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 15.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 87.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.25, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.25, "pitch": 83, "instrument": "piano"}, {"onset": 0.25, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.41, "offset": 0.56, "pitch": 7 ...[clipped]
- ✓healthy after requests
- 3passed7h 29m into the runtook 6m 25s
startup 15.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 15.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 88.4 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.25, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.25, "pitch": 83, "instrument": "piano"}, {"onset": 0.25, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.41, "offset": 0.56, "pitch": 7 ...[clipped]
- ✓healthy after requests
- 4passed8h 5m into the runtook 3m 14s
startup 16.5 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 16.5 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 86.2 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.25, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.25, "pitch": 83, "instrument": "piano"}, {"onset": 0.25, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.41, "offset": 0.56, "pitch": 7 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Final state:
Submitted — /workspace/output, 56 files, 387.6 MB (limit 1 GB), digest 7840691e…, 4th self-check PASSED:
- startup 16.5 s of 300 s
- real-time factor 94.3× (required 50×; 271 clips of 30 s in 86.2 s vs. the 162.6 s budget)
POST /transcribeandGET /healthboth green
What is served: the public YourMT3+ YPTF+Multi checkpoint used as published (no weight training of any kind), with two inference-side stages — decoding only the nine scored channels (0–8) instead of all 13, and a channel-level instrument-class correction that checks each decoding channel's class against harmonic-profile templates rendered from FluidR3, overriding the model only when 60 % of the channel's notes vote against it (the one fitted number, on a 0.55–0.70 plateau).
Measured: 0.3756 on the 10 provided example clips, honest leave-one-clip-out estimate 0.3517, against a 0.3002 uncorrected baseline; class-agnostic onset F1 0.63.
This session's work was a second round of analysis into why the remaining loss sits where it does, all of it rejected and recorded in logs/model_selection.txt §3:
- Decoding all 13 channels showed the model emits exactly one program per channel and never uses the mallets channel — so the mallets class is unreachable by construction, and the organ is decoded into the brass/reed channels.
- Templates rebuilt from the example clips themselves score 0.3640 LOCO (inside ten-clip noise) and still cannot separate the organ channel (margin 0.16) from the piano channel that must be protected (margin 0.52).
- Per-note classifiers (timbre features at the correct 16 kHz, note durations, channel+pitch rules) reach only 0.32 per-note accuracy against the model's own 0.64 — the mappings are properties of the clip, not the class. A 16 kHz resampling bug in the experiment loader was found and fixed; the shipped code never used those features.
- Masked kNN, class-balanced median distance and z-scored soft channel classification all land at 0.30–0.37, none better than the shipped vote.
One caveat worth stating plainly: the organ is 401 of the 3357 reference notes and still scores 0.000, and the mallets 45 notes also score 0.000 — a further ~0.05–0.10 of macro F1 is sitting in those two classes and I did not find a signal that reaches them.