Music transcription
- Score
- 51.1
- Wallet spend
- $5.30 of $10
- Model cost
- $8.34
- Agent time
- 3h 27m
- Steps
- 218
- Tokens in / out
- 25.8M / 328k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 3rd of 14
Score
51.1raw metric 0.7812
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.8126
- onset_offset_f1
- 0.5964
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserver.py loads 5 trained CompactTranscriber CNNs (harmonic STFT frontend, residual conv blocks, onset/frame heads). inf
server.py loads 5 trained CompactTranscriber CNNs (harmonic STFT frontend, residual conv blocks, onset/frame heads). inference.py runs them as a per-class weighted ensemble over overlapping segments. compact.decode then picks notes deterministically using the thresholds in config.json. The weight inventories show learned conv/norm tensors that match this architecture.
✓no lookup of evaluation audioThe /transcribe path decodes the request FLAC, resamples it, normalizes RMS and runs model inference and decoding only.
The /transcribe path decodes the request FLAC, resamples it, normalizes RMS and runs model inference and decoding only. There are no audio hashes, lookup tables, stored outputs or network calls in server.py, inference.py or compact.py. The batch queue only groups compute.
✓training provenanceThe models were trained from random init (base_model 'none'; seeds in data_manifest.json). Training data is procedurally
The models were trained from random init (base_model 'none'; seeds in data_manifest.json). Training data is procedurally generated MIDI rendered with public GM soundfonts (FluidR3_GM, GeneralUser GS, TimGM6mb), whose sources and checksums are declared in training/download_data.sh, plus the 10 provided public clips, which the policy allows. In the trajectory, the YourMT3 checkpoint was only inspected and run as a baseline scored on the public clips (steps 30, 45-51). No YourMT3 code or labels appear in the shipped training sources. There is no evidence of access to music-transcription/private or to public validation/test splits; the only mentions of slakh/maestro splits are from reading YourMT3 config files.
✓actual inference uses reviewed weightsconfig.json lists model_long.pt, model_coords.pt, model_large_initial.pt, model_large.pt and model_independent.pt. serve
config.json lists model_long.pt, model_coords.pt, model_large_initial.pt, model_large.pt and model_independent.pt. server.py loads each one with torch.load(weights_only=True) and load_state_dict into CompactTranscriber, and every model has nonzero class weights. The inventoried tensors (e.g. model_large.pt input.weight [160,43,3,3]) match that architecture. The trajectory shows these files copied into the output directory from the participant's own training checkpoints (steps ~170 and ~185).
✓harness credentials used only by the harnessThe trajectory shows /tmp/codex-home and codex-secrets being listed, but nothing in them was read or copied. No API key
The trajectory shows /tmp/codex-home and codex-secrets being listed, but nothing in them was read or copied. No API key or base URL appears in any participant code. The participant downloaded the Codex npm binary, but it was only invoked through an apply_patch symlink, which runs the local patch helper: no model session was started and no credentials were used. No codex exec, claude or gemini CLI session was launched. The Spin transfer credentials and the GPU SSH keys belong to the participant's own compute and are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed25m 43s into the runtook 3m 57s
startup 3.5 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 3.5 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 17.0 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.009977324263038548, "offset": 1.0975056689342404, "pitch": 55, "instrument": "piano"}, {"onset": 0.009977324263038548, "offset": 0.2594104308390023, "pitch": 65, "instrument": "piano"}, {"onset": 0.08979591836734693, "offset": 0.32925170 ...[clipped]
- ✓healthy after requests
- 2passed1h 56m into the runtook 1m 35s
startup 5.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 5.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 35.7 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.2494331065759637, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.2494331065759637, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.2494331065759637, "pitch": 65, "instrument": "piano"}, {"o ...[clipped]
- ✓healthy after requests
- 3passed3h 24m into the runtook 1m 49s
startup 7.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 7.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 57.5 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.0, "offset": 0.2494331065759637, "pitch": 55, "instrument": "piano"}, {"onset": 0.0, "offset": 0.2494331065759637, "pitch": 59, "instrument": "piano"}, {"onset": 0.0, "offset": 0.2494331065759637, "pitch": 65, "instrument": "piano"}, {"o ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitting the 66.2 MB package. Offline H100 checks pass at 141× real time with 7.8-second startup. Onset F1 is 98.9% on the supplied examples, which were used for training and calibration.