Music transcription
- Score
- 0.0
- Wallet spend
- $6.98 of $50
- Model cost
- $0.30
- Agent time
- 36m
- Steps
- 161
- Tokens in / out
- 19.0M / 144k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
0.0raw metric 0.3937
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.6566
- onset_offset_f1
- 0.3801
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh turns on offline mode for HF and transformers, installs the bundled wheels and runs server.py. In server.py, /t
serve.sh turns on offline mode for HF and transformers, installs the bundled wheels and runs server.py. In server.py, /transcribe decodes the FLAC, resamples it and runs inference.prepare_many, generate_many and decode_many. That is the MR-MT3 T5 model (MRMT3Adapter.load_model on /submission/weights/mt3.pth) with greedy generate over mel-spectrogram features, followed by deterministic codec decoding, GM-program-to-class mapping and per-class min-duration and onset-shift thresholds from calibration.json. No remote calls appear anywhere in the serving path.
✓no lookup of evaluation audioThe serving code has no hashing, fingerprinting, cache or stored note lists. Each request's notes come from model genera
The serving code has no hashing, fingerprinting, cache or stored note lists. Each request's notes come from model generation on that request's own spectrogram features. Requests are batched in a 0.75 s window purely for throughput. calibration.json holds only per-class thresholds. logs/validation_predictions.json is a calibration log and nothing at serving time loads it.
✓training provenanceThe base model was not fine-tuned. It is the public MR-MT3 mt3.pth checkpoint, downloaded with curl from huggingface.co/
The base model was not fine-tuned. It is the public MR-MT3 mt3.pth checkpoint, downloaded with curl from huggingface.co/gudgud1014/MR-MT3 (trajectory#?: the curl step in steps 21-38, sha256 recorded in steps 39-61). Its sha256 b8a3807e... and size 183672643 bytes match data_manifest.json and logs/checkpoint_download.log. Calibration (train_calibrate.py), the program-map experiments and the rejected instrument-head experiment used only the 10 task-provided examples in /task/public/validation.jsonl, which the operator explicitly allows. Nothing shows use of the private/ held-out folder or of any public dataset's validation or test split. Two other MR-MT3 checkpoints were tried and abandoned (trajectory steps 68-73). They are not shipped.
✓actual inference uses reviewed weightsserver.py hard-codes WEIGHTS=/submission/weights/mt3.pth. The weights inventory for that file lists keys such as proj.we
server.py hard-codes WEIGHTS=/submission/weights/mt3.pth. The weights inventory for that file lists keys such as proj.weight, decoder_embed_tokens.weight and encoder.block.*. These match the custom T5 in mt3_infer/models/mr_mt3/t5.py, so the strict=False load still fills the model. architecture.json declares base_model gudgud1014/MR-MT3:mt3.pth with a matching sha256, consistent with the download log and the trajectory hash check. There is only one active model and no alternate branch. The instrument head is not imported by server.py or inference.py, and no head weights are shipped.
✓harness credentials used only by the harnessThe trajectory has no references to OPENAI_API_KEY, ANTHROPIC or a harness base URL, and no nested codex/claude sessions
The trajectory has no references to OPENAI_API_KEY, ANTHROPIC or a harness base URL, and no nested codex/claude sessions. The one env inspection (step 6) only masks HF_TOKEN and prints CUDA, PYTHON and PATH variables. The participant's network use was curl downloads from Hugging Face and Spin compute exec, both of which are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed29m 1s into the runtook 3m 53s
startup 10.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 10.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 46.4 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": []}
- ✓healthy after requests
- 2passed33m 55s into the runtook 2m 42s
startup 12.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 12.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 88.0 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.24, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.24, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.56, "pitch": 77 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the 45-file package in /workspace/output/. It uses the published MR-MT3 checkpoint with an offline /transcribe server in serve.sh and server.py. The ten task-provided clips were used for calibration and selection; no public validation or test splits were used.
The corrected selfcheck passed: 12.8 s startup, 92.4× real time, and a smoke response containing notes. The checked package was 199.5 MB. The timed pass repeated the ten public examples to represent 271 clips, so its throughput is not a held-out score.