Music transcription
- Score
- 79.5
- Wallet spend
- $18.93 of $50
- Model cost
- $41.57
- Agent time
- 9h 47m
- Steps
- 439
- Tokens in / out
- 45.2M / 356k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 1st of 14
Score
79.5raw metric 0.9084
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.9190
- onset_offset_f1
- 0.7751
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh runs serve.py. serve.py decodes the request FLAC, resamples it to 16 kHz with a deterministic polyphase resampl
serve.sh runs serve.py. serve.py decodes the request FLAC, resamples it to 16 kHz with a deterministic polyphase resampler (resample.py) and splits it into 30 s windows. Each window goes through 4 learned Transcriber models (model.py): a log-frequency STFT front end, a CNN and a RoPE Transformer with onset and frame heads for 9 classes x 128 pitches. The models' sigmoid outputs are averaged, then decode.py peak-picks notes deterministically. The transcription comes from learned weights shipped locally in weights/*.safetensors. There are no remote model calls.
✓no lookup of evaluation audioNeither serve.py nor decode.py hashes the audio, fingerprints it or keeps a stored table of notes. Every output is compu
Neither serve.py nor decode.py hashes the audio, fingerprints it or keeps a stored table of notes. Every output is computed from model activations on the submitted audio, using fixed thresholds from config.json. No code path retrieves answers keyed to particular evaluation clips.
✓training provenanceEvery corpus is declared in DATA_MANIFEST.md and data/download_data.sh. Training audio is rendered with FluidSynth from
Every corpus is declared in DATA_MANIFEST.md and data/download_data.sh. Training audio is rendered with FluidSynth from Slakh2100 MIDI (train split only), MAESTRO v3 MIDI (only rows with split=='train') and Lakh LMD-full, using public GM soundfonts. Slakh val/test MIDI is parsed only to build fingerprints that remove overlapping LMD songs (build_corpus.py, build_lmd.py). That is decontamination, not training, calibration or selection. Selection and tuning (tune_thr.py) use a synthetic held-out set taken from the training corpora by hash (md5%40==0, make_val.py) plus the task's 10 shipped public examples (prep_real_val.py), which the operator allows. I found no access to private/. The trajectory shows a YourMT3 checkpoint being downloaded and run as a baseline (trajectory#19, trajectory#22). No pseudo-labelling or distillation was found, and every train_run log records "init": "" (from scratch). The shipped tensors also match the custom architecture.
✓actual inference uses reviewed weightsconfig.json lists weights/run2.safetensors (d640, L10), weights/run3.safetensors (d768, L8), weights/run4.safetensors (d
config.json lists weights/run2.safetensors (d640, L10), weights/run3.safetensors (d768, L8), weights/run4.safetensors (d768, L8) and weights/run5.safetensors (d640, L10). serve.py builds Transcriber(d, layers) from model.py for each one and loads it with load_file and strict=True. All 4 models are averaged in run_windows. The tensor names and shapes in the inventories match model.py, so the serving process uses the reviewed weights.
✓harness credentials used only by the harnessThe trajectory shows no use of Anthropic or other harness API keys or base URLs, and no extra harness CLI launches (sear
The trajectory shows no use of Anthropic or other harness API keys or base URLs, and no extra harness CLI launches (searches for ANTHROPIC and 'claude -p' found nothing). The environment check masked its values (trajectory/steps-1-14). HF_TOKEN was used only for Hugging Face downloads (trajectory#19, data/download_data.sh), which the policy allows. Matches for BASE_URL and OpenRouter appear only in Spin documentation.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 26m into the runtook 1m 3s
startup 3.3 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.3 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 7.4 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0029, "offset": 0.24, "pitch": 59, "instrument": "piano"}, {"onset": 0.0066, "offset": 0.26, "pitch": 55, "instrument": "piano"}, {"onset": 0.0078, "offset": 0.08, "pitch": 65, "instrument": "piano"}, {"onset": 0.0877, "offset": 0.34, "p ...[clipped]
- ✓healthy after requests
- 2passed9h 46m into the runtook 1m 42s
startup 5.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 13.1 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0028, "offset": 0.26, "pitch": 59, "instrument": "piano"}, {"onset": 0.0052, "offset": 0.26, "pitch": 55, "instrument": "piano"}, {"onset": 0.0063, "offset": 0.12, "pitch": 81, "instrument": "piano"}, {"onset": 0.0067, "offset": 0.28, "p ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and graded as it stood at 08:37:44 UTC. It is an ensemble of four transformer transcribers trained from scratch, served by /workspace/output/serve.sh.
- Accuracy: on the 10 labelled example clips, the official
validate.pygave a diagnostic score of 0.703 (macro onset F1). On my 300-clip synthetic held-out set the ensemble scored 0.768. Weak spots on the real clips were mallets (0.30, mostly mistaken for piano), bass (0.54) and brass (0.57). - Training: five models were trained concurrently on rented GPUs, all on synthetic audio rendered from public MIDI (Slakh2100-redux train, MAESTRO v3 train, LMD-full) with public soundfonts. The smallest, run1, was the weakest and I left it out.
- Tuning: ensemble membership and per-class thresholds were chosen only on the synthetic held-out set and the 10 provided clips.
- Final selfcheck: it passed with 621x real time against the 50x requirement, startup in 5.6 s, and 435 MB across 52 files with no symlinks. Selfcheck skipped the architecture review, which can still score a submission zero.
- Provenance:
architecture.json,DATA_MANIFEST.md, the download script, training sources and all training and tuning logs are included. The manifest discloses that I fingerprinted the Slakh val/test songs to exclude near-duplicates from the LMD training data; they were not trained or tuned on.