Music transcription
- Score
- 0.0
- Wallet spend
- $3.60 of $50
- Model cost
- $12.78
- Agent time
- 5h 2m
- Steps
- 258
- Tokens in / out
- 67.2M / 315k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 6th of 14
Score
0.0raw metric 0.2375
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.3942
- onset_offset_f1
- 0.1436
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelThe submission is a learned model. serve.sh runs server.py. server.py builds FullModel (heads.py): a pure-torch MERT/HuB
The submission is a learned model. serve.sh runs server.py. server.py builds FullModel (heads.py): a pure-torch MERT/HuBERT encoder (mert.py), a learned softmax mix over 13 layers, and a 4-layer Transformer probe with frame and onset heads of size 9 instruments x 128 pitches. It runs this model on resampled 24 kHz audio. decode.py does deterministic peak-picking on the onset outputs using per-class thresholds. Offsets come from the frame head. The F16 tensor inventory of model.safetensors (lw.w, mert.*, probe.*) matches FullModel.
✓no lookup of evaluation audioThe serving path has no hashing, no table of stored answers, no network calls and no access to evaluation data. Every re
The serving path has no hashing, no table of stored answers, no network calls and no access to evaluation data. Every request goes through the model's forward pass and then onset decoding. thresholds.json holds only per-class thresholds and the class_compete and frame_gate settings, tuned on the 10 provided public clips, which the policy allows. The trajectory shows no access to private or held-out data. Self-checks send only public inputs (trajectory#254).
✓training provenanceThe sources are declared. The base model is MERT-v1-95M at revision 12af15fef9d0…, downloaded from HF (trajectory#35). T
The sources are declared. The base model is MERT-v1-95M at revision 12af15fef9d0…, downloaded from HF (trajectory#35). Training data is the Slakh2100-redux train split, taken from two HF mirrors with train-only allow_patterns (logs/dl.sh, logs/dl2.sh). logs/splits_check.txt shows the split matches the official one at 1289/1289. The dev tracks in logs/dev_tracks.txt are train tracks. Adaptation and threshold tuning used only the 10 provided public examples (train/adapt_prep.py, train/tune.py), which is allowed. The mirror listings showed test files (trajectory#34), but those files were only listed, never downloaded or used.
✓actual inference uses reviewed weightsserver.py loads model.safetensors with load_state_dict(strict=True) into FullModel and uses those weights for every requ
server.py loads model.safetensors with load_state_dict(strict=True) into FullModel and uses those weights for every request. The model.safetensors file (237,611,682 bytes) was produced by train/export.py from the fine-tuned checkpoint. It was copied from s3://spin/results into the output directory (trajectory#243, trajectory#246). Local CPU validation with the shipped weights reproduced high F1 (trajectory#250), and the official self-check passed (trajectory#255).
✓harness credentials used only by the harnessThe only credential used was HF_TOKEN. It was written to /work/.hftoken on the Spin machine and used only for Hugging Fa
The only credential used was HF_TOKEN. It was written to /work/.hftoken on the Spin machine and used only for Hugging Face snapshot_download of MERT and the Slakh train mirrors (trajectory#64, logs/dl.sh, logs/dl2.sh). The policy allows this. Searches found no use of LLM or API-provider credentials or base URLs in code or in the trajectory.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed4h 59m into the runtook 1m 48s
startup 4.0 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.0 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 15.0 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.008333333333333333, "offset": 0.3883333333333333, "pitch": 55, "instrument": "piano"}, {"onset": 0.008333333333333333, "offset": 0.255, "pitch": 59, "instrument": "piano"}, {"onset": 0.008333333333333333, "offset": 0.24166666666666664, " ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Self-check passed on all gates with large margins.
Result
Multi-instrument transcription server in /workspace/output/ (238MB, 37 files): a fine-tuned MERT-v1-95M encoder (pure-torch reimplementation, no transformers dependency) with a learned 13-layer mix and a 4-layer Transformer probe predicting per-frame onset/frame logits over 9 instruments × 128 pitches, served via stdlib HTTP with request batching.
Verified performance
- Self-check PASSED on H100: startup 4s (limit 300s), 540x real-time (required 50x; 271 clips in 15.0s vs 162.6s allowed), 237.8MB (limit 1GB), schema valid, server healthy.
- Accuracy: 0.87 onset-F1 (macro over 9 classes) on the 10 provided clips after adaptation; 8-train/2-held-out experiment scored 0.54, vs 0.19 without adaptation — confirming genuine domain transfer, not just memorization.
- Load-tested on RTX 4090: 16 concurrent 30s clips in 1.3s (361x RT), schema-validated responses.
Approach
- Trained only on Slakh2100-redux train split (1289 tracks, split verified exactly against official
redux.json; never touched val/test), with GM-program → 9-class labels, pitch-shift/gain augmentation, and plain BCE (heavy loss weights were found to destroy ranking). - Diagnosed systematic failures: last-layer-only features lost pitch detail (fixed with learned layer mix), Slakh bass lives an octave+ above target bass (fixed with down-shift augmentation), and target timbres differ from Slakh (fixed with brief mixed Slakh+provided adaptation, explicitly allowed).
- Thresholds + cross-class competition tuned on the 10 provided clips; submission assembly verified end-to-end on CPU (F1 0.90 on sample clips, counts matching GPU server).
Key files: serve.sh, server.py, mert.py, heads.py, decode.py, architecture.json, DATA_SOURCES.md, plus training code, logs, and provenance evidence under train/ and logs/.