Voice endpointing
- Score
- 0.0
- Wallet spend
- $4.95 of $10
- Model cost
- $33.48
- Agent time
- 8h 23m
- Steps
- 365
- Tokens in / out
- 87.3M / 237k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 13th of 14
The reviewer found a rule violation; see the Review tab.
Score
0.0
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✗causal frame probabilities
The served model looks ahead within each 32 s chunk, so this check fails. predict.py cuts the audio into non-overlapping 32 s chunks and runs each chunk through the WavLM frontend in one pass. In wavlm_causal.py the first conv layer of the feature extractor is GroupNormConvLayer, which uses nn.GroupNorm(out_dim, out_dim) on a (B, C, T) tensor. With one group per channel, the mean and variance are taken over the whole time axis of the chunk. Every frame's features are therefore normalized using statistics from audio up to about 32 s later in the same chunk. The served checkpoint contains this layer (state_dict.feature_extractor.conv_layers.0.layer_norm.* in models/m1/frontend.pt), and load_wavlm_from_export does not replace it. The code docstring calls the conv feature extractor 'causal by construction', but that is wrong. The causal attention mask, the left-padded positional conv and the causal hand-crafted features do not remove this leak. Future samples change earlier frames' probabilities, which breaks the rule that all preprocessing is causal at each frame.
✓learned causal modelThe model is trained, and the probabilities come from running it on the audio. serve.sh runs serve.py. serve.py calls pr
The model is trained, and the probabilities come from running it on the audio. serve.sh runs serve.py. serve.py calls predict.load_models, which builds a WavLM encoder from models/m1/frontend.pt and an EOTHead from models/m1/head.pt. predict_streams then runs both on the decoded audio from both channels. The weight inventories hold the full WavLM-base-plus and head tensors. training_src/finetune.py fine-tunes the top WavLM layers and the head end-to-end on otoSpeech crops. logs/ft.log records 420 training conversations, 2400 steps of decreasing loss, and checkpoints selected by dev score on the 8 public conversations. Together these support the claimed model and fine-tuning.
✓no lookup of evaluation audioThe serving path (serve.py, predict.py, wavlm_causal.py, model.py, handfeat.py) only runs neural inference on the receiv
The serving path (serve.py, predict.py, wavlm_causal.py, model.py, handfeat.py) only runs neural inference on the received audio. I found no fingerprints, stored transcripts, event timelines or recording-ID matching.
✓training provenanceThe training corpus is declared in architecture.json and data/DATA_MANIFEST.md: otoearth/otoSpeech-full-duplex-turn-104h
The training corpus is declared in architecture.json and data/DATA_MANIFEST.md: otoearth/otoSpeech-full-duplex-turn-104h at a pinned revision, which is TurnBench's public training split. data/download_data.py and training_src/extract_features.py contain the download commands, and the trajectory shows API and audio downloads from that repo. The 8 shipped public conversations were used for dev selection and for a leave-one-out fine-tune experiment (training_src/devft.py); the policy allows this. The trajectory shows the participant reading the turnbench GitHub source, including the DEV_DATASET constant in data.py, and a web search result naming mundo-ai/turn-benchmark-dev. I found no download of the dev release or its derivatives and no use of held-out conversation IDs.
✓harness credentials used only by the harnessThe trajectory shows remote compute through the Spin CLI and HF_TOKEN passed to the Spin machine for Hugging Face downlo
The trajectory shows remote compute through the Spin CLI and HF_TOKEN passed to the Spin machine for Hugging Face downloads; both are allowed. I found no harness API key or base-URL use in the participant's code, no copying of those credentials, and no extra agent CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed6h 31m into the runtook 4m 16s
startup 79.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 79.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.96635}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.98638}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.98347}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.97348}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 120 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.8 s) plus the other 19771 s at conversation 124's pace (617 s in 3.6 s, 171.7x real time). The two conversations alone: 155.2x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2failed7h 58m into the runtook 6m 21s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Installing collected packages: nvidia-cusparselt-cu12, mpmath, typing-extensions, sympy, setuptool
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Installing collected packages: nvidia-cusparselt-cu12, mpmath, typing-extensions, sympy, setuptools, pycparser, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, MarkupSafe, fsspec, filelock, triton, scipy, nvidia-cusparse-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, jinja2, cffi, soundfile, nvidia-cusolver-cu12, torch Successfully installed MarkupSafe-3.0.4 cffi-2.1.1 filelock-4.0.12 fsspec-2026.9.0 jinja2-3.1.6 mpmath-1.3.0 networkx-3.7 numpy-2.5.3 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 pycparser-3.0 scipy-1.18.1 setuptools-84.0.0 soundfile-0.14.0 sympy-1.14.0 torch-2.8.0+cu128 triton-3.4.0 typing-extensions-4.16.0 File "/submission/serve.py", line 116 log(f"inference failed: {e!r} ^ SyntaxError: unterminated f-string literal (detected at line 116)
- 3error8h 12m into the runtook 6m 52sfree (stopped before any GPU time)
startup 79.3 s of 300 s; The self-check could not finish (BridgeUnavailable: The trusted relay could not run the HTTP bridge; the submission was not consulted: unknown exchange).; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold fir
2 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 79.3 s
- 4passed8h 19m into the runtook 3m 4s
startup 81.3 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 81.3 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.96635}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.98638}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.98347}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.97348}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 115 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.6 s) plus the other 19771 s at conversation 124's pace (617 s in 3.5 s, 178.5x real time). The two conversations alone: 160.7x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
All checks pass. Confirming the submission.