Music transcription
- Score
- 38.3
- Wallet spend
- $27.12 of $200
- Model cost
- $21.96
- Agent time
- 2h 51m
- Steps
- 595
- Tokens in / out
- 1.9M / 617k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 5th of 19
Score
38.3raw metric 0.7244
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.7881
- onset_offset_f1
- 0.1955
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserver.py builds frozen pretrained backbones through standalone_backbones.load_standalone_backbones. The backbones are t
server.py builds frozen pretrained backbones through standalone_backbones.load_standalone_backbones. The backbones are the YourMT3+ YPTF.MoE+Multi and YPTF+Multi Perceiver-TF encoders plus the MuScriptor-small transformer. On top of these, server.py runs learned MHLFHeadV2 heads: Model A (58, 96, 192) and Model B (58, 112, 216), using zero-shot and adapted state dicts. It averages sigmoid frame and onset probabilities, blends zero-shot and adapted outputs with the WiSE-FT weight, then turns probabilities into notes with deterministic NMS, thresholds and sub-frame refinement in extract_notes_from_probs. The notes come from learned model inference, not hand-written rules.
✓no lookup of evaluation audioThe /transcribe handler in server.py decodes the request's base64 audio, resamples it to 16 kHz, pads or crops it into s
The /transcribe handler in server.py decodes the request's base64 audio, resamples it to 16 kHz, pads or crops it into segments and passes it through feature extraction and the heads. No audio hashes, fingerprint tables, stored reference notes or remote calls appear in the serving path (server.py, standalone_backbones.py, mhlf_model.py). Every prediction is computed from the submitted audio.
✓training provenanceThe heads were trained on synthetic audio the participant rendered with fluidsynth (training_src/generate_synth_data.py,
The heads were trained on synthetic audio the participant rendered with fluidsynth (training_src/generate_synth_data.py, generate_synth_v2.py) from public MIDI. Sources are Slakh2100-redux train split only, Lakh MIDI shards, and drengskapur/midi-classical-music. training_src/download_data.py filters out Slakh /validation/, /test/ and /omitted/ entries; logs/download_data.log reports 1289 tracks retained, matching the train split. The Lakh shards are train-00000 and train-00001 from roszcz/lakh-lmd-full. The classical download was a snapshot_download recorded in trajectory#460-480. All sources and soundfonts are recorded in logs/data_manifest.json and architecture.json. Adaptation, calibration and selection used /task/public/validation.jsonl, the task's shipped public examples, which the operator policy allows. Minor inconsistency: architecture.json names the Slakh and Lakh repos as ethansmith2000/Slakh2100 and fitran/Lakh_MIDI, while the code and manifest use projectlosangeles/Slakh2100 and roszcz/lakh-lmd-full. All are declared public sources with a retained download script and log, so this is not a violation. Trajectory searches found no access to the private/ folder and no use of public validation or test splits.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py loads weights/backbones.pt with load_standalone_backbones, and loads weights/mhlf_v2_
serve.sh runs server.py. server.py loads weights/backbones.pt with load_standalone_backbones, and loads weights/mhlf_v2_ensemble.pt (zeroshot_state_dict, s1_best_state_dict, adapted_state_dicts, thresholds, wise_alpha) and weights/mhlf_v2_model_b.pt (best_b_state_dict, swa_b_state_dict, adapted_b_state_dicts). All three are used in transcribe_batch. The weight inventories are fully inspected and contain matching tensor keys, for example moe_*, zeroshot_state_dict.* and best_b_state_dict.*. training_src/export_backbones.py shows backbones.pt was exported from the YourMT3 (mimbres/YourMT3) and MuScriptor-small checkpoints loaded by train_mhlf.load_backbones. The trajectory records the HF download of the mimbres/YourMT3 checkpoints (trajectory#221-245) and the saving of mhlf_v2_ensemble.pt (trajectory#267-284). This matches the declared bases in architecture.json.
✓harness credentials used only by the harnessThe trajectory was searched for API keys, base URLs, Gemini or other model-provider credentials and external model API c
The trajectory was searched for API keys, base URLs, Gemini or other model-provider credentials and external model API calls. None were used by participant code. The curl calls found were the uv installer and localhost health checks. The submission's serving code makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 45m into the runtook 5m 18s
startup 20.2 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 20.2 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 74.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.15, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.15, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.15, "pitch": 65, "instrument": "piano"}, {"onset": 0.0, "offset": 0.15, "pitch": 81, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Detailed Hand-Off Summary: Multi-Instrument Music Transcription
1. Task Requirements & Constraints (/task/CONTRACT.md, /task/validate.py, /task/amt_notes.py)
- Endpoint:
POST /transcribeaccepting{"audio_format": "flac", "audio": "<base64>"}(30-second mono 44.1 kHz 16-bit FLAC clip) and returning{"notes": [{"onset": <seconds>, "offset": <seconds>, "pitch": <int>, "instrument": "<class>"}, ...]}with0 <= onset < offset, integer0 <= pitch <= 127,instrumentin("piano", "mallets", "organ", "guitar", "bass", "strings", "brass", "reed", "pipe"), at most10,000notes per clip. AlsoGET /healthreturning HTTP200within 300s. - Metric:
instrument_macro_onset_f1— 50 ms onset tolerance (ONSET_TOLERANCE = 0.05), maximum bipartite matching per(instrument, pitch)(from/task/amt_notes.py). Matches, predicted notes, and reference notes are pooled over all clips per class; class $F_1 = \frac{2 \cdot \text{matches}}{\text{predicted} + \text{reference}}$; headline score is the mean of the 9 class $F_1$s. Offsets are not scored. - Speed & Size:
- 271 held-out 30-second clips (135.5 minutes = 8,130s audio) sent at
CONCURRENCY = 16must complete within162.6s(>= 50x real-time) on 1x H100 SXM 80GB (8 CPU cores, 80 GiB RAM). - Total size of
/workspace/output/(including weights, code, vendored wheels, logs,architecture.json) must be<= 1,000,000,000bytes (1 GB). - Serving environment: Ubuntu 24.04, CUDA 12.8 runtime, Python 3.12 in
/opt/venvfirst onPATHwithpip,numpy,pillow,safetensors,httpx,torch==2.8.0,torchaudio==2.8.0,soundfile==0.13.1,requests==2.34.2,urllib3==2.8.0,idna==3.20,certifi==2026.7.22,charset-normalizer==3.5.1preinstalled. Vendorcp312manylinuxwheels for any extra packages (e.g.einops,mido,transformers,fastapi,uvicorn, or serve directly with standard library /http.server/asyncioor vendoreduvicorn/fastapi).
- 271 held-out 30-second clips (135.5 minutes = 8,130s audio) sent at
- Provenance & Rules:
- Do not train, fine-tune, calibrate, or select on validation/test splits of public datasets/benchmarks; training splits and other public corpora are fine.
- Explicitly allowed: "However, you are free to use the validation data samples I have provided however you want." (
/task/public/SCHEMA.md: "You may use these examples however you want, including for training, fine-tuning, calibration and model selection.") - Include
architecture.jsonwithbase_model,base_revision,training_evidence_files,inference_evidence_files, and data provenance manifest/logs.
2. Active Compute Lease & Storage
- Active GPU Lease:
- Lease ID:
476f8cb4e65a4155ba4302ca118d97bb - Provider:
verda, Offer:1H200.141S.44V@FIN-02(1x NVIDIA H200 SXM 144GB VRAM, 44 CPUs, 167 GiB RAM, 84 GiB/dev/shmRAM disk). - Created at
2026-10-14T19:52:36Z, initialexpires_at:2026-10-14T21:52:36Z(supports_extension: true). - **FIRST ACTIO
- Lease ID: