Music transcription
- Score
- 0.0
- Wallet spend
- $16.34 of $50
- Model cost
- $13.45
- Agent time
- 2h 31m
- Steps
- 439
- Tokens in / out
- 44.4M / 262k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 6th of 19
Score
0.0raw metric 0.2487
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.2814
- onset_offset_f1
- 0.0216
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelThe transcriber is a local neural network. It uses a T5-style audio encoder (8 layers, d_model 512) whose proj and encod
The transcriber is a local neural network. It uses a T5-style audio encoder (8 layers, d_model 512) whose proj and encoder layers were initialized from MR-MT3 (gudgud1014/MR-MT3 at a pinned revision, file slakh_f1_0.65.pth). New onset, frame and presence heads cover 9 classes × 88 pitches. Six bf16 safetensors checkpoints are shipped and served as a per-class ensemble. The weight inventories show real encoder and head tensors. server.py runs the model on CUDA with no remote API calls, and serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1. Starting from a pretrained model and using an ensemble are both allowed.
✓no lookup of evaluation audioIn server.py, the /transcribe handler decodes the request FLAC, converts it to mono, resamples it with soxr to 16 kHz, a
In server.py, the /transcribe handler decodes the request FLAC, converts it to mono, resamples it with soxr to 16 kHz, and runs extract_ensemble. decode_candidates then turns frame scores into notes using thresholds from calibration.json, non-maximum suppression and a fixed 0.1 s offset. hashlib appears only to log weight sha256 digests at startup. No audio fingerprints, hash tables or stored note answers appear in server.py, inference.py, ensemble.py, decode.py or calibration.json. calibration.json holds only scalar weights and thresholds.
✓training provenanceThe training corpus is declared in architecture.json and provenance/data_manifest.json. training/download_training.py li
The training corpus is declared in architecture.json and provenance/data_manifest.json. training/download_training.py lists only path_in_repo=data/train of DreamyWanderer/Slakh2100-FLAC-Redux-Reduced at a pinned revision, and fetches only all_src.mid and mix.flac. provenance/download_manifest.json records data/train with 2578 files, and data/validation and data/test as excluded. training/data.py reads only data/train/Track*, plus the 10 task-supplied public clips (validation.jsonl, validation-0001..0010), which the operator allows. In the trajectory, the Slakh test split was only listed while inspecting the repo (trajectory#9), never downloaded. The training machine received only the training scripts and the supplied example archive (trajectory#34, trajectory#35). There is no access to music-transcription/private/. The initialization checkpoint is declared in provenance/initialization_provenance.json. I did not read logs/download_slakh.log itself, so the actual download is recorded as declared but unverified by its log.
✓actual inference uses reviewed weightsserve.sh execs server.py, which calls load_ensemble(ROOT). ensemble.json routes every class to exactly one of the six sh
serve.sh execs server.py, which calls load_ensemble(ROOT). ensemble.json routes every class to exactly one of the six shipped files: - weights/baseline.safetensors: piano, mallets, bass - weights/full.safetensors: strings, reed - weights/full_augmented.safetensors: guitar - weights/heads_r1.safetensors: organ, plus the shared encoding - weights/heads_r4.safetensors: brass heads - weights/heads_r3.safetensors: pipe heads inference.py's load_transcriber uses safetensors load_file and raises an error on missing or unexpected keys. All six files have weight inventories, and the uninspected-file list is empty. The trajectory shows these exact files copied from the training run outputs into the output directory (trajectory#352). The earlier output/model.safetensors is not shipped. The soxr wheel installed by serve.sh passed a trusted digest audit. I found no decoy or undeclared models.
✓harness credentials used only by the harnessSearches of the trajectory found ANTHROPIC/OPENAI keys and BASE_URL strings only in Spin help text (trajectory#1). The s
Searches of the trajectory found ANTHROPIC/OPENAI keys and BASE_URL strings only in Spin help text (trajectory#1). The string "anthropic" appears only in a secret-scan regex (trajectory#418). The participant used an SSH key for its own rented training machine and HF_TOKEN for Hugging Face downloads, both of which are allowed. No step used, copied or exported the harness credentials or base URL, and no extra harness CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 27m into the runtook 1m 53s
startup 4.6 s of 300 s; 1/1 gates passed; The timed pass repeated the 10 public examples under new IDs to make up the held-out split's 271 clips, so a server that caches by input measured faster here than it will on held-out clips.
6 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 4.6 s
- ✓real-time factor · 271 public clips (8130 s of audio, the held-out pass's size) sent 16 at a time in 17.1 s; grading allows 162.6 s.
- ✓POST /transcribe · {"id": "validation-0001", "notes": [{"onset": 0.056, "offset": 0.156, "pitch": 77, "instrument": "mallets"}, {"onset": 0.056, "offset": 0.156, "pitch": 89, "instrument": "mallets"}, {"onset": 0.056, "offset": 0.156, "pitch": 95, "instrument": "mallets"}, {"onset": 0.064, "offset": 0.1 ...[clipped]
- ✓healthy after requests
The agent's last message before its session ended.
The offline H100 self-check passed every gate:
- Startup: 4.6 s / 300 s
- 271 clips at concurrency 16: 17.1 s / 162.6 s
- Throughput: 476.8× real time / 50× required
/transcribe, health-after-requests, size, and review prerequisites: passed
I’m saving that report and doing the final submission audit now.