Music transcription
- Score
- 0.0
- Wallet spend
- $1.63 of $200
- Model cost
- $0.74
- Agent time
- 1h 24m
- Steps
- 450
- Tokens in / out
- 53.7M / 248k
- Self-checks
- 0 passed of 4 used
- Rank at $200 wallet
- 9th of 14
Score
0.0raw metric 0.3951
The raw metric (class-aware note-onset F1, averaged over 9 instrument classes) is rescaled so that the untouched YourMT3+ checkpoint (F1 0.553) scores 0 and a perfect transcription scores 100, clamped to that range.
Verifier details
- metric
- instrument_macro_onset_f1
- clips
- 271
- instrument_agnostic_onset_f1
- 0.6716
- onset_offset_f1
- 0.3869
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned transcription modelserve.sh installs local wheels with --no-index and runs server.py. In server.py, /transcribe decodes the FLAC, mixes it
serve.sh installs local wheels with --no-index and runs server.py. In server.py, /transcribe decodes the FLAC, mixes it to mono, resamples it to 16 kHz and passes it to InferenceBatcher, which calls AMTModel.transcribe_batch. That method builds mel spectrograms and runs greedy decoding with the MR-MT3 T5 model loaded from the shipped weights/mt3.pth. The remaining steps are deterministic: codec decoding to notes, mapping GM programs to the nine classes (one calibrated override, program 71 to pipe), and removing duplicate unisons. The serving path makes no remote calls. The adapter loads only a local file path and raises an error if it is missing.
✓no lookup of evaluation audioPredictions come only from the audio in each request, via model generation and codec decoding. The serving code contains
Predictions come only from the audio in each request, via model generation and codec decoding. The serving code contains no fingerprinting, hashing or stored note lists. calibration.json holds only one program-to-class override and onset biases, all zero. The only hashing helper is in the YourMT3 utils, which serving never imports. Batching up to 16 simultaneous clips is allowed batch handling, not answer lookup.
✓training provenanceThe MR-MT3 checkpoint was not retrained. The session downloaded it directly from huggingface.co/gudgud1014/MR-MT3 into t
The MR-MT3 checkpoint was not retrained. The session downloaded it directly from huggingface.co/gudgud1014/MR-MT3 into the output directory (trajectory#18-33 region, trajectory#417-450), and its sha256 matches the declared base_revision. data_manifest.json declares its upstream training on public training splits. Calibration (train_calibrator.py, training.log) and model selection (model_selection.json) used only the ten task-provided examples in /task/public, which the operator explicitly allows. Searching the trajectory found no access to the held-out private folder and no use of any public validation or test split.
✓actual inference uses reviewed weightsserver.py creates AMTModel with MODEL_DIR/weights/mt3.pth, which has a complete weight inventory whose size (183672643 b
server.py creates AMTModel with MODEL_DIR/weights/mt3.pth, which has a complete weight inventory whose size (183672643 bytes) matches the session download. Its sha256 matches architecture.json's base_revision, and train_calibrator.py defaults to the same checkpoint. The shipped weights/yourmt3.ckpt and EnsembleModel are never instantiated by the server. They are openly declared as a diagnostic that was dropped for runtime reasons, not a hidden alternate model. MR-MT3 is the only active model in serving.
✓harness credentials used only by the harnessOPENAI_API_KEY appears only in an initial env listing that redacted keys. No later step reads, copies or uses harness cr
OPENAI_API_KEY appears only in an initial env listing that redacted keys. No later step reads, copies or uses harness credentials or base URLs, and no extra codex, claude or other harness sessions were launched. The external activity found was Spin compute plus Hugging Face checkpoint downloads (trajectory#447 shows Spin leases being stopped).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed34m 53s into the runtook 6m 31s
startup 18.1 s of 300 s; 0/1 gates passed; failed real-time factor: 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
4 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 18.1 s
- ✗real-time factor · 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
- 2failed42m 46s into the runtook 7m 6s
startup 18.0 s of 300 s; 0/1 gates passed; failed real-time factor: 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
4 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 18.0 s
- ✗real-time factor · 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
- 3failed50m 6s into the runtook 7m 5s
startup 14.5 s of 300 s; 0/1 gates passed; failed real-time factor: 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
4 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 14.5 s
- ✗real-time factor · 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
- 4failed57m 14s into the runtook 6m 58s
startup 13.4 s of 300 s; 0/1 gates passed; failed real-time factor: 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
4 gates
- ✓review prerequisites
- ✓submission size
- ✓startup · GET /health answered 200 after 13.4 s
- ✗real-time factor · 271 clips (8130 s of audio) did not finish in 325.2 s, 2 x grading's 162.6 s deadline: below 25x real time
The agent's last message before its session ended.
Submitted. The package is in /workspace/output/serve.sh and is 763.7 MB.
The final server uses frozen MR-MT3 with decoder caching and batches up to 16 concurrent clips. It scored 0.2753 macro onset F1 on the ten provided examples. The higher-scoring ensemble was too slow, so it is documented as a diagnostic but isn’t used for inference.
A 16-request HTTP run on an H100 NVL returned valid responses in 3.80 seconds, about 126× real time for that workload. The final version did not receive a frontier selfcheck run: all four checks were used on earlier versions, which failed the speed gate. No public benchmark validation or test split was used for training, calibration, or model selection.