Music transcription
- Score
- 0.0
- Wallet spend
- $2.70 of $50
- Model cost
- $1.11
- Agent time
- 6h 16m
- Steps
- 533
- Tokens in / out
- 49.4M / 397k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 6th of 19
Score
0.0raw metric 0.4253
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.5064
- onset_offset_f1
- 0.1223
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelThe shipped server runs a trained neural transcriber. serve.sh starts server.py, which loads model.pt through infer.load
The shipped server runs a trained neural transcriber. serve.sh starts server.py, which loads model.pt through infer.load_model. That function builds the UNetAMT CNN from model.py (about 7.49M parameters) on top of the deterministic LogSpec features from features.py. Transcription is a forward pass that produces sigmoid onset and frame probabilities for 9 classes × 128 pitches. Notes come from per-class thresholds, NMS peak picking and sustain offsets. The model.pt inventory has only model.* tensors, and they match the model.py modules exactly (enc1-5, up/mrg, head, wf [128,388,128], wc_onset/wc_frame [128,9], b_onset/b_frame [9,128]). There are no extra tables or stored outputs.
✓no lookup of evaluation audioserver.py decodes the request audio with soundfile and a torchaudio resample, micro-batches requests in the Batcher, and
server.py decodes the request audio with soundfile and a torchaudio resample, micro-batches requests in the Batcher, and calls infer.transcribe_batch. Nothing in that path hashes the audio, fingerprints it, looks it up in a table of stored answers, or makes a network call. The output comes only from model probabilities and decoding. Early in the session (trajectory#40-48) the participant matched pitch sequences and chroma between the public example clips and a few Slakh tracks. That was exploratory analysis of permitted public examples. Nothing from it ships, and it is not a serving-time lookup.
✓training provenanceThe shipped training data is declared and limited to the Slakh2100 train split. - Declared source: architecture.json and
The shipped training data is declared and limited to the Slakh2100 train split. - Declared source: architecture.json and DATA_MANIFEST.md name the Slakh2100-FLAC-Redux-Reduced HF dataset, train split only, re-rendered with the FluidR3_GM and TimGM6mb soundfonts. Calibration uses the public example clips (allowed). Internal selection uses every 25th train track (train.py val_base). - Train-only prep: slakh_prep.py defaults to --splits train, and the trajectory shows it run with --splits train (trajectory/steps-126-142). logs/prep.log reports 1289 tracks. The prep3 render is built from the same train MIDI (trajectory#175 region, steps-169-194). - Lineage: r3 is trained from scratch with no --init (trajectory#175, trajectory#260). r4 is initialized from r3/last.pt and r5 from r4/last.pt, both on prep3 (trajectory/steps-256-282, steps-317-339). r6 is initialized from r5/last.pt (trajectory/steps-413-440). logs/train_r3.log shows 1237 train and 52 val tracks. - No overlap: logs/overlap.log and logs/provenance_checks.txt report 0/10 public clips matching the train MIDI. - Validation-split exposure: the full HF snapshot also contains the validation/test folders. Early exploration downloaded a few validation-split mixes (trajectory#42, trajectory#43), and a 4-step CPU smoke-test dataset included Track01632 (trajectory#89, trajectory#90, trajectory#103). That smoke test was abandoned and is not an ancestor of the shipped weights, so the shipped weights' provenance is untainted. - Verification gap: the download command is recorded in DATA_MANIFEST.md, but I could only partly confirm the source by retained logs. I record it as declared, partially verified provenance.
✓actual inference uses reviewed weightsserve.sh runs server.py with the default --ckpt HERE/model.pt. infer.load_model calls torch.load, rebuilds UNetAMT from
serve.sh runs server.py with the default --ckpt HERE/model.pt. infer.load_model calls torch.load, rebuilds UNetAMT from the checkpoint config, and runs load_state_dict(ckpt['model']), which is strict by default. The reviewed model.pt inventory matches model.py exactly. transcribe_batch applies these loaded models to the request features. The ONSET_THRS values in serve.sh match the thresholds declared in architecture.json. No other weights or alternate inference path appear in the serving call graph.
✓harness credentials used only by the harnessTrajectory searches for API key and harness credential patterns (API_KEY, ANTHROPIC, private-data paths) found no step w
Trajectory searches for API key and harness credential patterns (API_KEY, ANTHROPIC, private-data paths) found no step where the participant read or used harness credentials. The participant used the provided spin compute tooling and its SSH key to run GPU training, and there is no sign of credential misuse. The serving code (serve.sh, server.py) makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 21m into the runtook 6m 31s
startup 2.2 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.2 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 11.7 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 3.06, "offset": 3.3200000000000003, "pitch": 38, "instrument": "bass"}, {"onset": 19.0, "offset": 19.06, "pitch": 43, "instrument": "bass"}, {"onset": 26.8, "offset": 28.880000000000003, "pitch": 55, "instrument": "guitar"}, {"onset": 19.9 ...[clipped]
- ✓healthy after requests
- 2passed1h 56m into the runtook 8m 14s
startup 2.7 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.7 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 14.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 2.44, "offset": 3.3, "pitch": 64, "instrument": "piano"}, {"onset": 3.06, "offset": 3.4, "pitch": 38, "instrument": "bass"}, {"onset": 18.68, "offset": 19.86, "pitch": 67, "instrument": "piano"}, {"onset": 29.16, "offset": 29.54, "pitch": ...[clipped]
- ✓healthy after requests
- 3passed3h 58m into the runtook 4m 9s
startup 2.4 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.4 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 13.5 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 27.44, "offset": 28.6, "pitch": 59, "instrument": "piano"}, {"onset": 2.44, "offset": 3.46, "pitch": 64, "instrument": "piano"}, {"onset": 18.68, "offset": 19.8, "pitch": 67, "instrument": "piano"}, {"onset": 2.44, "offset": 3.0, "pitch": ...[clipped]
- ✓healthy after requests
- 4passed6h 14m into the runtook 1m 52s
startup 2.1 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.1 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 10.6 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 2.44, "offset": 3.48, "pitch": 64, "instrument": "piano"}, {"onset": 27.44, "offset": 28.580000000000002, "pitch": 59, "instrument": "piano"}, {"onset": 1.8, "offset": 3.48, "pitch": 64, "instrument": "piano"}, {"onset": 19.62, "offset": 2 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the submission:
Model — from-scratch U-Net AMT (7.5 M params, 388 log-freq bins, 16 kHz, 50 fps) with separate onset and sustain heads over 9 instrument classes × 128 pitches, trained only on the Slakh2100 train split (1289 tracks / 88.9 h) rendered three ways (released mix + FluidR3_GM + TimGM6mb SoundFont re-renders). Shipped checkpoint: r6 step 17500, selected on a 52-track internal train-split holdout (macro onset F1 0.3014).
Decoding — per-class onset thresholds calibrated on the 10 provided clips, offsets from the sustain head, NMS 2 frames, no gate/suppression.
Verification
- Public-clip official metric: 0.3878 (instrument-agnostic onset F1 0.518)
- Cross-validated check (thresholds fit on 5 clips, scored on the other 5): 0.3787, vs 0.3037 for a single global threshold and 0.3208 for the oracle global — so the calibration generalises rather than memorising the examples
- Local
validate.py: 0.3857, 0 failed requests selfcheck(4/4 used, last one on the exact submitted folder): 37 files / 30.4 MB, startup 2.1 s, RTF 767× real time (needs 50×), POST 0.964 s, healthy after load
Provenance — architecture.json + DATA_MANIFEST.md name every corpus with download commands and licences; logs/ carries the download, preprocessing, rendering, training (r3–r6), calibration, fold-generalisation, adaptation-rejection and overlap-check logs. The fine-tuning-on-public-clips route was tested and rejected (it overfit: 0.95 in-sample vs 0.12 on held-out clips), so the shipped weights come from the Slakh train split alone; the public clips were used only for threshold calibration, which the task explicitly permits.