Music transcription
- Score
- 1.6
- Wallet spend
- $12.18 of $200
- Model cost
- $51.61
- Agent time
- 2h 58m
- Steps
- 112
- Tokens in / out
- 76.3M / 150k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 8th of 14
Score
1.6raw metric 0.5601
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.7270
- onset_offset_f1
- 0.5377
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh starts src/server.py. Each /transcribe request decodes the FLAC and sends it to Engine.transcribe_many in src/e
serve.sh starts src/server.py. Each /transcribe request decodes the FLAC and sends it to Engine.transcribe_many in src/engine.py. The engine runs the vendored YourMT3+ (YPTF.MoE+Multi) network, built by inference_loader.py from weights/model.ckpt, on that audio: spectrogram, then encoder, then greedy decoding with a KV cache. Its output tokens are detokenized and mapped from GM programs to the 9 classes with deterministic post-processing. The local patches (MoE grouping on device, cache-pruned batched decoding, a transformers-5 cache shim) change how fast the model runs, not what it computes. The source contains no remote calls; for example, requests is set to None in utils.py.
✓no lookup of evaluation audioThe server and engine compute predictions only from the decoded request waveform. No audio hashing or fingerprinting, st
The server and engine compute predictions only from the decoded request waveform. No audio hashing or fingerprinting, stored transcriptions, answer tables or access to private or evaluation data appears in the serving path. The only hashlib import is in the vendored utils.py, which is unused at inference. Selection used only the 10 /task/public clips (trajectory#83 shows 10 rows), which the policy allows.
✓training provenanceNo training or calibration was done. The weights are the published mimbres/YourMT3 checkpoint at pinned revision e45ebd7
No training or calibration was done. The weights are the published mimbres/YourMT3 checkpoint at pinned revision e45ebd70398682d54b7bb1901a5216e18f3b1824. architecture.json and logs/provenance.json declare its sha256, and logs/download_log.txt records the huggingface-cli download command, the sha256/size and the copy to weights/model.ckpt. The download itself has no independent proof beyond that self-reported log and the hash match, so it is recorded as unverified. Checkpoint selection used only the task's 10 public clips. The URMP-Reduced train split was used only for a diagnostic and does not feed the shipped weights. No validation, test or private split was used.
✓actual inference uses reviewed weightsserve.sh exports CHECKPOINT=weights/model.ckpt, which src/engine.py passes to load_model_for_inference. inference_loader
serve.sh exports CHECKPOINT=weights/model.ckpt, which src/engine.py passes to load_model_for_inference. inference_loader.py rebuilds YourMT3 from the checkpoint hparams and loads the state_dict. strict=False skips only pitchshift.* keys, and the trajectory runs log missing=0 unexpected=0, so no weights are left randomly initialized. trajectory#96 confirms the shipped weight file's hash matches the declared checkpoint sha256. There is one model and no decoy or alternate weight path. The pinned transformers wheel passes the trusted binary audit.
✓harness credentials used only by the harnessSearching the trajectory for the private set, API keys, base URLs and model-provider credentials found no use or copy of
Searching the trajectory for the private set, API keys, base URLs and model-provider credentials found no use or copy of harness credentials. curl appears only in local health checks. GPU runs used the participant's own compute tooling, and the submission was made through the normal flow (trajectory#110).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 51m into the runtook 3m 46s
startup 23.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 23.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 94.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.08, "offset": 0.24, "pitch": 77, "instrument": "piano"}, {"onset": 0.08, "offset": 0.24, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.86, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.55, "pitch": 77 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The transcription server is submitted. It serves the published YourMT3+ multi-instrument checkpoint and returns every pitched note with onset, offset, MIDI pitch, and one of the nine instrument classes.
On the 10 labeled clips the note-onset score is 0.274, the same as the published checkpoint. The official check started in 24 seconds and transcribed the full evaluation-sized pass at 86× real time (the limit is 50×). The submission is 588 MB, under the 1 GB cap.