Music transcription
- Score
- 2.6
- Wallet spend
- $9.45 of $10
- Model cost
- $54.17
- Agent time
- 10h 17m
- Steps
- 484
- Tokens in / out
- 95.6M / 227k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 8th of 14
Score
2.6raw metric 0.5647
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.5973
- onset_offset_f1
- 0.2551
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs server.py. server.py loads every weights/*.pt through model.load_crnn, a CRNN with 4 conv blocks, an FC la
serve.sh runs server.py. server.py loads every weights/*.pt through model.load_crnn, a CRNN with 4 conv blocks, an FC layer, a 2-layer BiGRU and frame/onset/offset/regression heads over 9 classes x 128 pitches. It computes a log-mel spectrogram (features.py), averages the sigmoid outputs of the 3 checkpoints, and turns them into notes with peak picking and the tuned thresholds. The fp16 tensor inventories of weights/model.pt, model2.pt and model3.pt match the CRNN's parameter names and shapes, so this is a learned neural transcription model.
✓no lookup of evaluation audiotranscribe() in server.py only reads the audio, resamples it, extracts features, runs the networks and decodes notes. Th
transcribe() in server.py only reads the audio, resamples it, extracts features, runs the networks and decodes notes. There is no hashing, filename or fingerprint lookup, stored answer table or network call. thresholds.json holds only scalar decoding thresholds. The trajectory shows no access to music-transcription/private: the only 'private' hit is an HF metadata field ('private': false, trajectory#13).
✓training provenanceDATA_MANIFEST.md and architecture.json declare the training data: the Slakh2100 redux train split from HF DreamyWanderer
DATA_MANIFEST.md and architecture.json declare the training data: the Slakh2100 redux train split from HF DreamyWanderer/Slakh2100-FLAC-Redux-Reduced, plus fluidsynth GM-soundfont re-renders of those train MIDIs. The soundfonts come from apt and HF and are recorded in setup.log and sf2_dl.py. download.sh uses allow_patterns ['data/train/*'], download.log is retained, and dataset.py reads only data/train. The participant's session did list the dataset's test files (trajectory steps 1-19) but only downloaded data/train/*. Model selection in train.py and threshold tuning in tune.py use only the public shipped validation clips, which the operator allows. Later edits added speed/pitch augmentation and class balancing (trajectory#291) without changing the data sources. The shipped evidence/dataset.py is a snapshot from before those edits, a minor mismatch that does not affect provenance. Training starts from scratch (base_model none). No held-out or public validation/test split was used.
✓actual inference uses reviewed weightsThe serving call graph runs serve.sh, then server.py, which calls model.load_crnn on each sorted weights/*.pt and loads
The serving call graph runs serve.sh, then server.py, which calls model.load_crnn on each sorted weights/*.pt and loads ck['model'] into the CRNN. Those are exactly the three inventoried checkpoints, whose tensor shapes match the CRNN definition in model.py and the training code in evidence/train.py. The trajectory shows these checkpoints being copied from the Spin training runs into output/weights and the final file listing before submission (trajectory#481). The server uses no other model or external service.
✓harness credentials used only by the harnessSearches of the trajectory for API_KEY, BASE_URL and credential use found only Spin CLI help text. The participant used
Searches of the trajectory for API_KEY, BASE_URL and credential use found only Spin CLI help text. The participant used Spin GPU machines and spin s5cmd storage, which the operator allows, and HF downloads, which are allowed. No harness credentials were used for the participant's own model calls or outside the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed3h 6m into the runtook 4m 54s
startup 3.0 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.0 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 21.1 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0888, "offset": 0.2396, "pitch": 71, "instrument": "piano"}, {"onset": 0.0891, "offset": 12.4364, "pitch": 83, "instrument": "piano"}, {"onset": 0.2406, "offset": 0.2699, "pitch": 43, "instrument": "bass"}, {"onset": 0.2428, "offset": 0. ...[clipped]
- ✓healthy after requests
- 2passed5h 44m into the runtook 1m 53s
startup 3.0 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.0 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 18.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0883, "offset": 0.5409, "pitch": 77, "instrument": "piano"}, {"onset": 0.0887, "offset": 0.1162, "pitch": 78, "instrument": "piano"}, {"onset": 0.0887, "offset": 0.42, "pitch": 83, "instrument": "piano"}, {"onset": 0.2445, "offset": 19.5 ...[clipped]
- ✓healthy after requests
- 3passed9h 36m into the runtook 7m 51s
startup 4.5 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.5 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 61.8 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0883, "offset": 0.2404, "pitch": 77, "instrument": "piano"}, {"onset": 0.0885, "offset": 0.1595, "pitch": 83, "instrument": "piano"}, {"onset": 0.2444, "offset": 0.2715, "pitch": 55, "instrument": "piano"}, {"onset": 0.2446, "offset": 1. ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
All files verified. Submitting: