Music transcription
- Score
- 0.0
- Wallet spend
- $13.96 of $50
- Model cost
- $29.39
- Agent time
- 3h 37m
- Steps
- 832
- Tokens in / out
- 3.2M / 739k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 6th of 19
Score
0.0raw metric 0.542
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.6951
- onset_offset_f1
- 0.4620
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserver.py transcribes with three learned neural models: the MuScriptor medium LM transcriber (checkpoints/muscriptor_med
server.py transcribes with three learned neural models: the MuScriptor medium LM transcriber (checkpoints/muscriptor_medium/model.safetensors), YourMT3+ YPTF+Multi fine-tuned on public data (checkpoints/ymt3_ptf_multi_ft_bf16.pt) and YourMT3+ YPTF.MoE+Multi (checkpoints/ymt3_moe_multi_bf16.pt). A learned NoteInstrumentMLP (checkpoints/note_instrument_mlp.pt) fuses their outputs. The MLP class in note_features.py hardcodes c_w1 as [9, 93, 24], which matches the shipped tensor inventory, so the training script's hidden_dim=64 argument is not a mismatch. The weight inventories show real tensors for all four checkpoints.
✓no lookup of evaluation audioThe /transcribe handler in server.py decodes the request audio and runs it through the models and the decode_predictions
The /transcribe handler in server.py decodes the request audio and runs it through the models and the decode_predictions post-processing. It does no hashing, fingerprint matching, stored-answer lookup or network calls, and serve.sh sets offline environment variables. A trajectory search found no access to music-transcription/private.
✓training provenanceSources are declared in architecture.json: - Base models: cocktailpeanut/muscriptor-medium @27246ba6 and mimbres/YourMT3
Sources are declared in architecture.json: - Base models: cocktailpeanut/muscriptor-medium @27246ba6 and mimbres/YourMT3 @e45ebd70. - Training data: only the data/train files of the DreamyWanderer Slakh/URMP/MusicNet repos (training/download_train_data.py; logs/download_train_data.log shows 130 train tracks), plus the 10 permitted public validation examples. The trajectory shows test/validation folders were only listed in repo metadata, never downloaded for training (trajectory#100). The MoE and MuScriptor checkpoints are bf16 conversions of the public HF snapshots (trajectory#739, trajectory#740). The fine-tuned YPTF and the MLP come from the shipped training scripts, and their logs match (logs/train_ymt3.log, logs/train_note_classifier.log).
✓actual inference uses reviewed weightsserve.sh runs server.py, which loads checkpoints/muscriptor_medium/model.safetensors, checkpoints/ymt3_ptf_multi_ft_bf16
serve.sh runs server.py, which loads checkpoints/muscriptor_medium/model.safetensors, checkpoints/ymt3_ptf_multi_ft_bf16.pt, checkpoints/ymt3_moe_multi_bf16.pt and checkpoints/note_instrument_mlp.pt from the submission directory. All of them are active in transcribe_batch_wavs. These are the reviewed weight files, copied into the output directory from the training machine (trajectory#743). No remote weights or API calls are used at serve time.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENAI, GEMINI, API_KEY, BASE_URL, printenv, claude and codex found no use or copying
Trajectory searches for ANTHROPIC, OPENAI, GEMINI, API_KEY, BASE_URL, printenv, claude and codex found no use or copying of harness credentials and no extra harness CLI launches. Only HF_TOKEN was used, for Hugging Face downloads, which the policy allows (trajectory#100, trajectory#739). The SSH key and compute-session credentials used for the remote GPU are not harness model credentials. No shipped code references harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 21m into the runtook 5m 48s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 680, in <module> main(
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 680, in <module> main() File "/submission/server.py", line 672, in main engine = TranscriptionEngine(ROOT_DIR / "checkpoints", device="cuda") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 161, in __init__ self.ymt3_ft, self.tm_ft, self.tie_ft = _build_ymt3_model( ^^^^^^^^^^^^^^^^^^ File "/submission/server.py", line 50, in _build_ymt3_model from model.init_train import update_config File "/submission/src/YourMT3/amt/src/model/init_train.py", line 15, in <module> import pytorch_lightning as pl ModuleNotFoundError: No module named 'pytorch_lightning'
- 2passed3h 32m into the runtook 2m 58s
startup 17.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 17.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 103.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.074, "offset": 0.384, "pitch": 77, "instrument": "piano"}, {"onset": 0.074, "offset": 0.384, "pitch": 83, "instrument": "piano"}, {"onset": 0.224, "offset": 0.544, "pitch": 43, "instrument": "piano"}, {"onset": 0.384, "offset": 0.544, "p ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Hand-off Summary: Multi-Instrument Music Transcription
1. Task Requirements & Constraints (CONTRACT.md)
- Goal: Transcribe 30-second mono 44.1 kHz 16-bit FLAC clips into pitched notes:
- Request:
POST /transcribewith JSON{"audio_format": "flac", "audio": "<base64-encoded-flac-bytes>"}(also supports raw FLAC bytes). - Response:
{"notes": [{"onset": <seconds>, "offset": <seconds>, "pitch": <int 0..127>, "instrument": "<class>"}, ...]}(0 <= onset < offset, max 10,000 notes/clip).
- Request:
- Nine Instrument Classes (General MIDI
0..79):piano(GM0..7), 2.mallets(GM8..15), 3.organ(GM16..23), 4.guitar(GM24..31), 5.bass(GM32..39), 6.strings(GM40..51excl. 46 harp / 47 timpani), 7.brass(GM56..63), 8.reed(GM64..71), 9.pipe(GM72..79).
- Metric: Note-onset F1 (50ms tolerance, same instrument class & MIDI pitch, Kuhn maximum bipartite matching via
amt_notes.py), pooled over all clips per class, then macro-averaged over the 9 classes (instrument_macro_onset_f1). Offsets are not scored. - Hardware & Runtime Gates:
- Evaluates offline on 1x H100 SXM 80GB (8 CPU cores, 80 GiB RAM, Ubuntu 24.04, CUDA 12.8 runtime without
nvcc, Python 3.12 in/opt/venvwithtorch==2.8.0,torchaudio==2.8.0,soundfile==0.13.1,numpy,safetensors,httpx,requestspreinstalled). bash /submission/serve.sh(copied from/workspace/output/) receivesHOST=0.0.0.0,PORT=8000,MODEL_NAME=submission.GET /healthmust return 200 within 300s.- Speed Gate: Must achieve >= 50.0x real-time at concurrency 16:
- On the 271 held-out clips (8,130s audio), concurrency 16 must finish within 162.6s.
- On the 10 public validation clips (
/task/public/validation.jsonl, 300s audio total) used byvalidate.pyandfrontier selfcheck, all 10 clips sent at concurrency 16 must finish within 6.00s (300s / 50 = 6.00s)!
- Size Gate:
/workspace/output/must be<= 1,000,000,000bytes (currently ~784 MB).
- Evaluates offline on 1x H100 SXM 80GB (8 CPU cores, 80 GiB RAM, Ubuntu 24.04, CUDA 12.8 runtime without
- Important: Never end a turn without a tool call until final submission is in
/workspace/output/andfrontier submit --confirmis run.
2. Active Spin Compute Lease & Paths
Active GPU Lease (H100 SXM 80GB on RunPod)
- Lease ID:
2796e3e6ceae4d51b5f945ce69a8fc7a - Expires At:
2026-10-14T22:54:55Z(in ~10 minutes! IMMEDIATELY extend on the very first turn:spin compute extend 2796e3e6ceae4d51b5f945ce69a8fc7a --seconds 3600 --request-id h100-ext-3). - SSH / SCP:
- Key:
/tmp/h100.key - SSH:
ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i /tmp/h100.key -p 17046 root@64.247.201.60 - SCP:
scp -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i /tmp/h100.key -P 17046 ...
- Key:
Complete Submission Package in /workspace/output/ (~784 MB total)
- [
/workspace/output/serve.sh](file:///worksp