Music transcription
- Score
- 17.8
- Wallet spend
- $31.65 of $200
- Model cost
- $37.59
- Agent time
- 5h 13m
- Steps
- 973
- Tokens in / out
- 98.0M / 675k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 7th of 19
Score
17.8raw metric 0.6326
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.7847
- onset_offset_f1
- 0.4804
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs server_mt3.py with model.pt. server_mt3.py loads the checkpoint's 'model' state dict into the vendored MR-
serve.sh runs server_mt3.py with model.pt. server_mt3.py loads the checkpoint's 'model' state dict into the vendored MR-MT3 T5ForConditionalGeneration with strict=True. It computes 512-bin mel features from the request audio, runs greedy cached decoding (mt3_fast.py), and decodes MT3 events into notes (mt3_events.py). The only rule-based step comes after the model: mt3_postprocess.py applies deterministic count-ratio gates that relabel organ and guitar/bass programs on the model's output. The transcription itself is learned. model.pt is 187.8 MB, which matches a model of about 45.9M fp32 parameters. The inventory inspector could not open it because of a PosixPath pickle global, which the training script's metadata also explains. That is not a violation.
✓no lookup of evaluation audioThe /transcribe handler in server_mt3.py decodes the base64 audio it receives, resamples it, and sends it through the ne
The /transcribe handler in server_mt3.py decodes the base64 audio it receives, resamples it, and sends it through the network. It does no hashing, fingerprinting, table lookup, file read of stored answers or network call. mt3_postprocess.py only relabels predicted notes using deterministic ratios. The trajectory shows no access to private or evaluation data.
✓training provenanceThe base checkpoint is the public kunato/mt3-pytorch mt3.pth. provenance/base_download_command.txt retains the curl comm
The base checkpoint is the public kunato/mt3-pytorch mt3.pth. provenance/base_download_command.txt retains the curl command and sha b8a3807e…, and logs/final_all/config.json records the same revision. data_manifest.json declares the training data with pinned HF revisions: Slakh train MIDI and audio, the NSynth train parquet shards, FluidSynth soundfont renders of Slakh train MIDI, and the 10 public examples the operator allows. The download scripts (training/download_slakh.py, download_slakh_audio.py, download_nsynth.py) fetch train splits only, and an explicit assertion blocks validation and test paths. Internal holdouts are drawn from train material. The trajectory does show 'test' and 'validation' entries, but only in remote file listings, not in anything that was downloaded. training/run_mt3_final_all.sh, train_mt3.py and mt3_data.py fine-tune from data/mt3/mt3.pth on these declared roots.
✓actual inference uses reviewed weightsserve.sh starts only server_mt3.py, pointing --model at $DIR/model.pt. server_mt3.py loads that file into the T5 network
serve.sh starts only server_mt3.py, pointing --model at $DIR/model.pt. server_mt3.py loads that file into the T5 network with strict=True and uses it for every request. The secondary-model branch is inactive. In trajectory#943, final_all.pt is copied to output/model.pt (sha 51d37bcb…, the selected run step_06000) and secondary_model.pt is removed. In trajectory#944, serve.sh is edited to serve a single model. architecture.json records the same selected checkpoint sha.
✓harness credentials used only by the harnessI searched the trajectory for ANTHROPIC, API_KEY, OPENROUTER, BASE_URL, printenv and claude -p, and none of them appear.
I searched the trajectory for ANTHROPIC, API_KEY, OPENROUTER, BASE_URL, printenv and claude -p, and none of them appear. The Agent call at step 10 is the harness's own subagent tooling, not the participant extracting or reusing credentials. The serving code makes no external API calls and runs with offline environment variables.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 54m into the runtook 2m 19s
startup 16.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 16.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 63.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.4, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.4, "pitch": 83, "instrument": "piano"}, {"onset": 0.25, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.41000000000000003 ...[clipped]
- ✓healthy after requests
- 2passed4h 55m into the runtook 7m 57s
startup 22.2 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 22.2 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 149.3 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.4, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.4, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.55, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.41000000000000003 ...[clipped]
- ✓healthy after requests
- 3passed5h 6m into the runtook 6m 21s
startup 16.4 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 16.4 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 67.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.4, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.4, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.41000000000000003 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-24 06:01:30 UTC.
- Model: Single all-corpus MR-MT3 checkpoint
- Package: 276 files, 347,822,225 bytes
- Final self-check: Passed
- Startup: 16.4 s
- 271 clips: 67.6 s
- Throughput: 120.198× real time
- Public diagnostic: Macro F1 0.960921, zero failed requests; this is an all-ten fitted in-sample diagnostic, not a held-out score.