Music transcription
- Score
- 0.0
- Wallet spend
- $9.91 of $10
- Model cost
- $1.90
- Agent time
- 1h 43m
- Steps
- 269
- Tokens in / out
- 27.9M / 69k
- Self-checks
- 1 passed of 3 used
- Rank at $10 wallet
- 10th of 19
Score
0.0raw metric 0.299
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.7423
- onset_offset_f1
- 0.3032
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh starts server.py, which loads the MuScriptor medium autoregressive transcription LM (an MT3-style transformer w
serve.sh starts server.py, which loads the MuScriptor medium autoregressive transcription LM (an MT3-style transformer with mel-spectrogram conditioning) from weights/model.safetensors. The /transcribe handler decodes the request audio, converts it to mono and resamples it to 16 kHz. It then runs greedy model._model.generate over 5-second chunks. Notes are produced by decode_model_tokens plus a fixed GM-group-to-class mapping. All other pre- and post-processing is deterministic. The download helpers are offline stubs, so serving makes no remote calls. The notes therefore come from a learned model run on the submitted audio.
✓no lookup of evaluation audioThe handler computes its output only from the audio it receives: decode, resample, run the model, decode tokens. There i
The handler computes its output only from the audio it receives: decode, resample, run the model, decode tokens. There is no fingerprinting, hashing, file-name matching, stored transcription table or remote lookup anywhere in server.py or the muscriptor package. Batching up to 32 concurrent jobs is just an inference optimization.
✓training provenanceNo training, fine-tuning, calibration or selection was done (training_log.txt, data_manifest.json). The weights are the
No training, fine-tuning, calibration or selection was done (training_log.txt, data_manifest.json). The weights are the public Hugging Face MuScriptor/muscriptor-medium checkpoint declared in architecture.json. It was downloaded at trajectory#171, cast to fp16 at trajectory#184 with no other change to the values, and copied into the submission at trajectory#185. The /task/public examples were used only for evaluation, which the operator allows. No access to the private/ held-out set was found. A tried-and-abandoned mr_mt3.pth checkpoint is not shipped.
✓actual inference uses reviewed weightsserver.py calls TranscriptionModel.load_model on the reviewed weights/model.safetensors, with weights/config.json alongs
server.py calls TranscriptionModel.load_model on the reviewed weights/model.safetensors, with weights/config.json alongside. That config (dim 1024, 16 heads, 24 layers, card 1395) matches the tensor inventory and the medium architecture. The load path uses load_file and load_state_dict with download_if_necessary stubbed to the local path. No alternate or decoy model files are loaded during serving.
✓harness credentials used only by the harnessThe trajectory and shipped code show no use of the harness API key or base URL. There are no Anthropic, OpenAI or OpenRo
The trajectory and shipped code show no use of the harness API key or base URL. There are no Anthropic, OpenAI or OpenRouter calls, and no 'spin openrouter key create' was run; the only match is the Spin help text at trajectory#93. HF_TOKEN was used only for Hugging Face downloads (trajectory#171, trajectory#174), which the operator allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 4m into the runtook 5m 52s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 8, in <module> from mu
3 gates
- ✓review prerequisites
- ✓submission size
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 8, in <module> from muscriptor import TranscriptionModel File "/submission/muscriptor/__init__.py", line 4, in <module> from muscriptor.transcription_model import TranscriptionModel File "/submission/muscriptor/transcription_model.py", line 19, in <module> import muscriptor.accelerator File "/submission/muscriptor/accelerator.py", line 10, in <module> from packaging.version import Version ModuleNotFoundError: No module named 'packaging'
- 2failed1h 11m into the runtook 17m 34s
startup 4.7 s of 300 s; 0/1 gates passed; failed real-time factor: 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 191.9 s; grading allows 162.6 s.; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.7 s
- ✗real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 191.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.4, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.4, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.56, "pitch": 77, ...[clipped]
- ✓healthy after requests
- 3passed1h 29m into the runtook 3m 48s
startup 2.8 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 2.8 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 75.9 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.09, "offset": 0.4, "pitch": 77, "instrument": "piano"}, {"onset": 0.09, "offset": 0.4, "pitch": 83, "instrument": "piano"}, {"onset": 0.24, "offset": 0.56, "pitch": 43, "instrument": "piano"}, {"onset": 0.4, "offset": 0.56, "pitch": 77, ...[clipped]
- ✓healthy after requests