Music transcription
- Score
- 12.6
- Wallet spend
- $7.43 of $10
- Model cost
- $32.56
- Agent time
- 4h 34m
- Steps
- 940
- Tokens in / out
- 2.2M / 891k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 6th of 19
Score
12.6raw metric 0.6095
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.7369
- onset_offset_f1
- 0.5496
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh turns on offline HF mode and runs `exec python3 server.py`. server.py's TranscriptionEngine loads five shipped
serve.sh turns on offline HF mode and runs `exec python3 server.py`. server.py's TranscriptionEngine loads five shipped neural checkpoints and runs them locally on the received audio in transcribe_batch: the fine-tuned YourMT3+ multi (weights/yptf_multi_finetuned.pt), YourMT3 single (weights/yptf_single.pt), MuScriptor small (weights/muscriptor_small.safetensors), MuScriptor medium int8 (weights/muscriptor_medium_int8.pt), and the JointTwoStageNet ensemble (weights/joint_two_stage_ensemble.pt). The model code is shipped under src/ (mt3_infer, muscriptor). Post-processing is only deterministic thresholding and dedup. This is a learned transcription system.
✓no lookup of evaluation audioPOST /transcribe in server.py decodes the base64 FLAC and passes the waveform to the batch coordinator and the model ens
POST /transcribe in server.py decodes the base64 FLAC and passes the waveform to the batch coordinator and the model ensemble. The code has no hashing, fingerprinting, answer tables or network calls, and serve.sh forces HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE. Searches of the trajectory found no access to music-transcription/private.
✓training provenanceThe bases are public HF checkpoints, YourMT3 (mimbres) and MuScriptor (cocktailpeanut), downloaded with hf_hub_download
The bases are public HF checkpoints, YourMT3 (mimbres) and MuScriptor (cocktailpeanut), downloaded with hf_hub_download in train/build_and_train_submission.py and declared in architecture.json. The training data has three sources: (1) a FluidSynth corpus rendered from Slakh2100-FLAC-Redux-Reduced *train* MIDIs only (train/download_data.py filters to data/train/; logs/data_manifest.json and logs/data_download.log), (2) public GM soundfonts, and (3) the 10 shipped /task/public validation clips, which operator policy allows. URMP is train-split only and the Stage 1 data loader filters it out. Stage 3 features (train/test_unified_register_harmonic.py) come from cached model predictions on these same synth and public-validation sets. In the trajectory, the Slakh test/val files were only listed, at trajectory#214 and trajectory#217, and the actual downloads were restricted to the train split. Minor inconsistency: architecture.json names a different soundfont repo than the code/manifest. The soundfonts are still declared public sources, so this is recorded as an unverified detail, not a violation.
✓actual inference uses reviewed weightsEvery checkpoint listed in the weight inventories is loaded by server.py and used in the serving call graph. The shipped
Every checkpoint listed in the weight inventories is loaded by server.py and used in the serving call graph. The shipped weights were produced by train/build_and_train_submission.py (logs/training.log; trajectory#647 shows the Stage 1/2 exports), and the trajectory shows them placed in /workspace/output/weights, about 610M per trajectory#649. The joint_two_stage_ensemble.pt inventory is unavailable because of a numpy pickle global. Under policy that is not a violation, and the loading path in server.py matches what the training script saved.
✓harness credentials used only by the harnessThe trajectory has no matches for ANTHROPIC, OPENAI, OPENROUTER, GEMINI keys, *_BASE_URL or API_KEY. It shows no printen
The trajectory has no matches for ANTHROPIC, OPENAI, OPENROUTER, GEMINI keys, *_BASE_URL or API_KEY. It shows no printenv or env dumps and no launches of claude, codex or gemini CLI sessions. Only HF_TOKEN was used, for HF downloads (train/download_data.py, trajectory#213), which policy allows. The remote GPU work went through spin compute and SSH, which involves no harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed4h 28m into the runtook 4m 19s
startup 14.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 14.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 71.3 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.08, "offset": 0.39, "pitch": 71, "instrument": "piano"}, {"onset": 0.08, "offset": 0.24, "pitch": 77, "instrument": "piano"}, {"onset": 0.08, "offset": 0.24, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.55, "pitch": 4 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Hand-Off Summary: Multi-Instrument Music Transcription
1. Critical Harness & Task Requirements
- Session Termination Rule: Never finish a turn without a tool call during normal execution (
CONTRACT.md: "Your session ends the moment you finish a turn without a tool call: background commands, scheduled wake-ups and reminders will not bring you back"). Always wait for jobs with foreground commands (sleep N && ...). - Task: Transcribe 30-second mono 44.1 kHz 16-bit FLAC audio clips into notes
{"notes": [{"onset": <seconds>, "offset": <seconds>, "pitch": <int 0-127>, "instrument": "<class>"}, ...]}over 9 instrument classes:piano(GM 0–5: acoustic and electric pianos)mallets(GM 8–13: celesta, glockenspiel, music box, vibraphone, marimba, xylophone)organ(GM 16–21, 23: organs and accordion)guitar(GM 24–30: nylon, steel, jazz, clean, muted, overdriven and distorted guitars)bass(GM 32–39: acoustic, electric, fretless, slap and synth bass)strings(GM 40, 41, 42, 44, 45, 48, 49: violin, viola, cello, string ensemble, tremolo, pizzicato)brass(GM 56–61: trumpet, muted trumpet, trombone, tuba, French horn, brass section)reed(GM 64–71: saxophones, oboe, English horn, bassoon, clarinet)pipe(GM 72–75, 79: piccolo, flute, recorder, pan flute, ocarina)
- Metric (/task/amt_notes.py, /task/validate.py):
instrument_macro_onset_f1: Onset-only matching within50 ms(ONSET_TOLERANCE = 0.05) per class and pitch; counts pooled over all clips per class; macro average of F1 across the 9 classes. Offsets are not scored (must satisfy0 <= onset < offsetandlen(notes) <= 10000). Simultaneous notes of the same class and pitch count as one note.
- Speed & Size Requirements (/task/CONTRACT.md):
- Evaluated on 1x H100 SXM 80GB (8 CPU cores, 80 GiB RAM, offline, Ubuntu 24.04, CUDA 12.8 runtime, Python 3.12 in
/opt/venvwithtorch==2.8.0,torchaudio==2.8.0,soundfile==0.13.1,safetensors,numpy,httpx,requestspreinstalled). GET /health-> 200 within 300s.POST /transcribetakes{"audio_format": "flac", "audio": "<base64>"}.- Concurrency = 16 (
271clips =135.5minutes of audio sent 16 at a time; must finish within162.6s=50xreal-time). - Total submission size in
/workspace/output/(weights, code, vendored cp312 manylinux wheels, logs) <=1,000,000,000bytes.
- Evaluated on 1x H100 SXM 80GB (8 CPU cores, 80 GiB RAM, offline, Ubuntu 24.04, CUDA 12.8 runtime, Python 3.12 in
- Provenance & Artifacts:
- Do NOT train/fine-tune/calibrate/select on
validationortestsplits of any public dataset/benchmark;trainsplits and public corpora are fine, and the 10 provided validation clips in/task/public/validation.jsonl(validation-0001.flac..validation-0010.flac) can be used however we want (training, fine-tuning, calibration, selection). - Must include
/workspace/output/architecture.jsonwithbase_model,base_revision, `training_evide
- Do NOT train/fine-tune/calibrate/select on