Music transcription
- Score
- 64.3
- Wallet spend
- $85.65 of $200
- Model cost
- $68.74
- Agent time
- 6h 40m
- Steps
- 311
- Tokens in / out
- 100.1M / 325k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 4th of 14
Score
64.3raw metric 0.8403
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.8781
- onset_offset_f1
- 0.7749
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh starts server.py with no network access: wheels are installed offline and HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE a
serve.sh starts server.py with no network access: wheels are installed offline and HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE are set. For each request, server.py decodes the FLAC, resamples it to 16 kHz and cuts it into 32767-sample segments. It then runs the batched learned YourMT3+ model in ymt_infer.py: a Perceiver-TF encoder with MoE (fastmoe.py patches the Mixtral MoE forward with equivalent maths) and a multi-channel T5 decoder doing greedy static-KV decoding. postproc.py turns the MT3 tokens into notes and maps GM programs to the 9 classes. The weight inventory of weights/model.safetensors lists about 47M F32 YourMT3 tensors (encoder and decoder blocks). Transcriptions come from model inference, not hand-written rules.
✓no lookup of evaluation audioThe request path in server.py and postproc.py (decode_request, resample, segment, tokens_for_segments, tokens_to_notes_m
The request path in server.py and postproc.py (decode_request, resample, segment, tokens_for_segments, tokens_to_notes_multi) has no hashing, fingerprinting, stored-answer table, cache keyed on input or remote call. Warmup uses random noise. The trajectory shows no access to music-transcription/private/. The selfcheck note about input caching is generic harness text and is not evidence that this server caches.
✓training provenanceThe base is the public mimbres/YourMT3 Space checkpoint at a pinned revision (architecture.json, training/download_data.
The base is the public mimbres/YourMT3 Space checkpoint at a pinned revision (architecture.json, training/download_data.sh). Fine-tuning data is FluidSynth renders of Slakh2100-redux MIDI. training/gen_data.py takes tracks only from splits['train'] in training/logs/slakh_redux_splits.json, and DATA_MANIFEST.json declares the MIDI tarball and split-listing HF repos with revisions and a sha256, plus the GM soundfont sources. Calibration and selection used only the 10 provided public validation clips, which the task and operator policy allow (trajectory#3), and synthetic sets rendered from train-split MIDI (training/eval_synth_lib.py, training/logs/final_selection.txt). No val/test split or private held-out data was used. The public downloads are declared with retained commands and logs, but I did not independently verify their byte-level contents.
✓actual inference uses reviewed weightsserver.py finds weights/model.safetensors and builds a single Transcriber. ymt_infer.py loads weights/config.json and mo
server.py finds weights/model.safetensors and builds a single Transcriber. ymt_infer.py loads weights/config.json and model.safetensors with safetensors.torch.load_file and load_state_dict, ignoring only the pitchshift keys. training/export.py writes this config.json/model.safetensors format, and training/eval_final.py produces soup_last2 as the weight average of r7s8000 and r8s8000, matching architecture.json. In trajectory#302 the participant copied /root/exp/soup_last2/model.safetensors and config.json into the submission with matching sha256 values (188,412,552 bytes, the same as the inventory). trajectory#303 then validated the exact submission folder ('1 model(s) loaded'). No decoy checkpoint or unshipped weights are used.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN was used only to download public HF assets, which the policy allows. In trajectory#304 the p
In the trajectory, HF_TOKEN was used only to download public HF assets, which the policy allows. In trajectory#304 the participant scanned the output for the token and found it in no file. Searches for ANTHROPIC, API_KEY, BASE_URL and spin chat found no use of harness model or API credentials by the participant's own code. serve.sh runs offline and needs no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 30m into the runtook 1m 27s
startup 11.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 11.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 31.2 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.05, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.05, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.05, "pitch": 65, "instrument": "piano"}, {"onset": 0.0, "offset": 0.05, "pitch": 81, " ...[clipped]
- ✓healthy after requests
- 2passed3h 44m into the runtook 4m 45s
startup 16.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 16.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 94.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.05, "pitch": 81, "instrument": "piano"}, {"onset": 0.0, "offset": 0.24, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.24, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.24, "pitch": 65, " ...[clipped]
- ✓healthy after requests
- 3passed6h 35m into the runtook 1m 24s
startup 11.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 11.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 27.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.05, "pitch": 81, "instrument": "piano"}, {"onset": 0.0, "offset": 0.24, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.24, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.24, "pitch": 65, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The folder matches what the final selfcheck verified (126 files, 207.9 MB). Confirming: