Voice endpointing
- Score
- 0.0
- Wallet spend
- $0.00 of $50
- Model cost
- $4.96
- Agent time
- 41m
- Steps
- 364
- Tokens in / out
- 48.9M / 72k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 8th of 14
Score
0.0raw metric 0.2764
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9842
- recall
- 0.2764
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 421
- fn
- 1102
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver.py sends both audio channels through a learned network: the pretrained DualTurn Endpointing ONNX model, loaded fr
server.py sends both audio channels through a learned network: the pretrained DualTurn Endpointing ONNX model, loaded from models/model.onnx with onnxruntime. Its weight inventory contains a learned network: a Mimi causal conv/transformer encoder, a projection, two LSTM layers and the eot/VAD heads. download_weights.py fetches that checkpoint from anyreach-ai/dualturn-endpointing at the pinned revision c3860ed7. The trajectory shows how it got there: the file was downloaded from HF into /workspace/cache (trajectory#44-68 range), the commit sha was looked up (trajectory#242), and the file was copied unmodified into output/models (trajectory#246, size 161,640,815 bytes, which matches the inventory). The served model is learned and used directly; there are no hand-coded rules.
✓causal frame probabilitiesThe model outputs a per-frame eot probability at 0.08 s hop (12.5 Hz). The Mimi encoder is causal, and the LSTM weights
The model outputs a per-frame eot probability at 0.08 s hop (12.5 Hz). The Mimi encoder is causal, and the LSTM weights have shape [1,1024,256], i.e. num_directions=1, so the LSTMs are unidirectional. server.py runs 40 s chunks. Each later chunk adds only 10 s of past context and keeps only the new frames. Trimming or padding uses the last past value. No future audio, bidirectional pass or smoothing goes into a frame's probability. The participant also tested chunk equivalence on random audio (trajectory#161-200 range).
✓no lookup of evaluation audioserver.py decodes the request audio, resamples it, runs the ONNX model and returns frame probabilities for each speaker
server.py decodes the request audio, resamples it, runs the ONNX model and returns frame probabilities for each speaker (predict(wav1,wav2) and predict(wav2,wav1)). It does not fingerprint or hash audio, keep stored labels, read the gold data, or look anything up. The only data the trajectory shows being used is /task/public/validation.jsonl, which holds the 8 shipped public conversations (ids 20, 21, 22, 94, 95, 122, 124, 125), and it was used only for local evaluation (trajectory#249, trajectory#201-240 range). Nothing from it is shipped into the server.
✓training provenanceThe participant did no training, calibration or selection on new data. The shipped model is the unmodified public DualTu
The participant did no training, calibration or selection on new data. The shipped model is the unmodified public DualTurn checkpoint. architecture.json declares its upstream training corpora as Switchboard (200h, training split) and OtoSpeech (289h, training split), which matches the HF model card and config the participant read in the trajectory. download_weights.py keeps the checkpoint download command. The upstream training data is unverified beyond the model card, but it is declared and public. Searches found no download of mundo-ai/turn-benchmark-dev or its derivatives, no load_dataset or HF dataset downloads, and no held-out ids. TurnBench dev appears only in web-search results (trajectory#32). The only local evaluation data was the allowed public conversations.
✓harness credentials used only by the harnessEvery match for OPENROUTER_API_KEY, or for the relay/litellm ports 4123/4125, appears in `ps -ef` output that shows the
Every match for OPENROUTER_API_KEY, or for the relay/litellm ports 4123/4125, appears in `ps -ef` output that shows the harness's own launcher wrapper (provider-pin relay, litellm proxy, `unset OPENROUTER_API_KEY`). None of these are participant-authored commands (trajectory#241, trajectory#249). Searches found no claude, codex or ANTHROPIC usage, no printenv/env dumps, and no participant calls to model APIs. HF_TOKEN appears only as an optional download header, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed36m 21s into the runtook 5m 19s
startup 5.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 5.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9958897829055786}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9940577745437622}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9941455125808716}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9920254945755005}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 3880 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 129.5 s) plus the other 19771 s at conversation 124's pace (617 s in 117.0 s, 5.3x real time). The two conversations alone: 5.3x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Solution Summary
-
Model Architecture & Causality:
- Deployed DualTurn Endpointing (
anyreach-ai/dualturn-endpointing, revisionc3860ed71210fe0144af35af340fd7a4dec3d2d3), built upon frozen Mimi speech features (kyutai/mimi) with a causal recurrent turn-taking readout. - The model is strictly causal: its prediction at frame $t$ depends exclusively on audio up through $(t+1) \cdot h$. Causality was verified both through architecture inspection and perturbation tests confirming future audio modifications produce exactly zero difference in past frame outputs.
- Deployed DualTurn Endpointing (
-
Inference & Serving Runtime:
- Created
server.pyandserve.shimplementingGET /healthandPOST /predict. - Vendored offline manylinux
cp312wheels foronnxruntime,onnxruntime-gpu,soundfile,cffi, and required dependencies inwheels/. serve.shinstalls the vendored wheels into the environment offline and starts the server with zero external network access.- Employed an overlapping sliding context window (40s chunk, 10s context) that runs with constant memory footprint and avoids $O(T^2)$ attention scaling on long conversational audio.
- Created
-
Performance & Verification:
- Tested locally against the evaluation harness
validate.py, achieving real-time factors exceeding $4.0\times$ real-time on CPU with strong recall at 5% FPR. - Evaluated using
frontier selfcheckon the H100 target environment:- Startup: 5.5s (allowed: 300s, 98% margin)
- Real-Time Factor: $5.27\times$ real-time (required: $\ge 3.0\times$, 76% margin)
- All Gates: Passed with HTTP 200 on all public test conversations and healthy status maintained.
- Final submission confirmed and submitted via
frontier submit --confirm.
- Tested locally against the evaluation harness