Music transcription
- Score
- 0.0
- Wallet spend
- $3.60 of $10
- Model cost
- $11.33
- Agent time
- 6h 53m
- Steps
- 547
- Tokens in / out
- 55.9M / 304k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 9th of 14
Score
0.0raw metric 0.4348
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.5265
- onset_offset_f1
- 0.1121
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs code/serve.py. Its load() builds a Transcriber from code/model.py for each model listed in weights/config.
serve.sh runs code/serve.py. Its load() builds a Transcriber from code/model.py for each model listed in weights/config.json: a CNN front end, an 8-layer TransformerEncoder, and onset and frame heads. It loads the checkpoint state dicts and runs every request's decoded audio through log-mel features and both networks. The onset logits are averaged, then decoded deterministically in code/infer.py using sigmoid, NMS, per-class thresholds and a time bias. The weight inventories show tensor names matching Transcriber, about 31.5M parameters each. There are no remote calls or rule-based transcription stand-ins.
✓no lookup of evaluation audioThe serving path (code/serve.py, code/infer.py) has no audio hashing, fingerprinting or stored-answer tables. weights/co
The serving path (code/serve.py, code/infer.py) has no audio hashing, fingerprinting or stored-answer tables. weights/config.json holds only model dimensions, thresholds, NMS and bias settings, and per-model mel mean/std. training/provenance_check.py is a development-only script. It compared the public example note lists with Slakh MIDI to identify where the examples came from. It is never imported by serving and its output is not used in inference. No access to the private set was found.
✓training provenanceDATA_MANIFEST.md and architecture.json declare the sources: the Slakh2100 train split from the public HF repo aashishbis
DATA_MANIFEST.md and architecture.json declare the sources: the Slakh2100 train split from the public HF repo aashishbishow/slakh2100_midi_only_redux, public GM soundfonts (fluid-soundfont-gm and wrightflyer/SF2_SoundFonts), and the 10 public example clips, used only for monitoring, selection and calibration, which policy allows. training/setup.sh and training/setup2.sh contain the download commands. The trajectory shows the training tracks dir being rebuilt from the train split only (trajectory#83-99: prepare_midi.py --slakh /workspace/data/slakh_git/train). The Slakh validation and test splits were downloaded (fetch_slakh.py default, setup2 git clone), but they were used only by the provenance fingerprint check and never for training, calibration or selection. No validation or test path was fed to prepare_midi. No --resume run was invoked, so initialization is from scratch (seeds 7 and 8). Exact download logs are not retained, so the data downloads are recorded as unverified, but every source is declared.
✓actual inference uses reviewed weightsweights/config.json lists models model.pt and model2.pt. code/serve.py loads exactly these files with torch.load(weights
weights/config.json lists models model.pt and model2.pt. code/serve.py loads exactly these files with torch.load(weights_only=True) into Transcriber, and the POST handler uses them for every request. Both files are in the inventory (pytorch, inspection complete, about 125.9MB each) with tensor names matching code/model.py. The trajectory shows training/finalize.py producing package_final from run2 and run1 model_best, which was validated end to end through serve.sh and copied into weights/ (trajectory#492, trajectory#494, trajectory#497, trajectory#498). base_model none matches from-scratch training.
✓harness credentials used only by the harnessThe trajectory shows only HF_TOKEN used for Hugging Face downloads (also via /root/.hftoken in training/setup2.sh and tr
The trajectory shows only HF_TOKEN used for Hugging Face downloads (also via /root/.hftoken in training/setup2.sh and training/fetch_slakh.py) and Spin compute/R2 session credentials used for Spin storage transfers. Both are allowed. Searches for ANTHROPIC, OPENAI, OPENROUTER, harness base URLs and claude -p CLI launches found no use of harness credentials in participant code and no extra harness CLI runs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed5h 34m into the runtook 4m 39s
startup 3.0 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.0 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 12.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.06, "offset": 0.36, "pitch": 55, "instrument": "piano"}, {"onset": 0.06, "offset": 0.36, "pitch": 65, "instrument": "piano"}, {"onset": 0.06, "offset": 0.36, "pitch": 71, "instrument": "piano"}, {"onset": 0.06, "offset": 0.36, "pitch": 7 ...[clipped]
- ✓healthy after requests
- 2passed5h 48m into the runtook 2m 24s
startup 3.5 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.5 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 15.5 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.06, "offset": 0.36, "pitch": 55, "instrument": "piano"}, {"onset": 0.06, "offset": 0.36, "pitch": 65, "instrument": "piano"}, {"onset": 0.06, "offset": 0.36, "pitch": 71, "instrument": "piano"}, {"onset": 0.06, "offset": 0.36, "pitch": 7 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
One provenance issue: the shipped finalize.py predates the wide-grid edit (it couldn't reproduce the shipped thresholds). Syncing the exact version that produced the config: