Voice endpointing
- Score
- 43.3
- Wallet spend
- $2.97 of $10
- Model cost
- $10.96
- Agent time
- 50m
- Steps
- 147
- Tokens in / out
- 18.8M / 63k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 7th of 14
Score
43.3raw metric 0.8043
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.8480
- recall
- 0.8043
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1225
- fn
- 298
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe model in production is a trained neural network. serve.sh sets EOT_WEIGHTS to weights/vap_oto.ckpt and runs server.p
The model in production is a trained neural network. serve.sh sets EOT_WEIGHTS to weights/vap_oto.ckpt and runs server.py. server.py builds VapGPT (frozen CPC encoder, GRU, causal downsample, GPT and GPTStereo transformers, VAP head) and loads the checkpoint with strict key checks. Scores come from the model's p_now run on the audio. The trajectory shows how the checkpoint was obtained: the public viks66/VAP_checkpoints oto.ckpt was downloaded and moved into the weights directory (trajectory#117).
✓causal frame probabilitiesEach frame's score uses only audio up to the end of that frame. The CPC conv stack looks only about 9 ms ahead, and the
Each frame's score uses only audio up to the end of that frame. The CPC conv stack looks only about 9 ms ahead, and the CConv1d downsample is padded causally, so 50 Hz frame m depends on samples before the frame boundary. ChannelNorm is per-timestep. All self- and cross-attention in vap/modules.py uses causal tril/ALiBi masks. Chunked inference in server.py (28 s windows, 8 s step) keeps only the trailing frames of each later chunk, so no output uses future audio. The post-processing (a sliding min of 1 - p_now) pads on the left and is causal.
✓no lookup of evaluation audioNothing in the serving path (server.py, vap/*) fingerprints, hashes or matches incoming audio against stored labels or r
Nothing in the serving path (server.py, vap/*) fingerprints, hashes or matches incoming audio against stored labels or recordings. Outputs are computed purely from model inference on the decoded audio.
✓training provenanceThe training and selection data are declared and allowed. The served checkpoint is the public VAP oto.ckpt, which per da
The training and selection data are declared and allowed. The served checkpoint is the public VAP oto.ckpt, which per data_manifest.json and architecture.json was trained on the otoSpeech full-duplex training corpus, TurnBench's declared training split. The participant's logged fine-tune (train_vap.py, download_log.txt, training_notes.txt) used only otoSpeech conversations downloaded via huggingface-cli, and it was not served. The 8 shipped public examples were used only to choose the operating point, which policy allows. The download of the served checkpoint is logged; the otoSpeech download command is retained but its source is unverified beyond the logs. The only contact with mundo-ai/turn-benchmark-dev was a metadata file listing with no download (trajectory#28). There is no use of the JSALT derivative, held-out ids or val/test splits. The upstream baseline README mentioning a dev threshold sweep (trajectory#44) is upstream exposure, not the participant's own use.
✓harness credentials used only by the harnessNo harness credentials or nested harness CLIs were used. Trajectory searches for API_KEY, BASE_URL, XAI, x.ai and printe
No harness credentials or nested harness CLIs were used. Trajectory searches for API_KEY, BASE_URL, XAI, x.ai and printenv found only Spin CLI setup documentation (trajectory#3 region, steps 1-11). There is no use of ANTHROPIC, OPENAI, OPENROUTER, GEMINI or xAI keys or base URLs, and no nested grok, claude or codex CLI was launched. HF access was used only for downloads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed47m 3s into the runtook 3m 29s
startup 74.3 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.3 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9987262487411499}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9985992312431335}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.998211681842804}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9972286820411682}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 200 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 6.7 s) plus the other 19771 s at conversation 124's pace (617 s in 6.0 s, 102.2x real time). The two conversations alone: 102.5x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. It serves a causal Voice Activity Projection model that scores when each speaker has released the floor.
On the eight labeled conversations, a causal 7-frame minimum over the release probability reaches about 0.75 recall at a 5% false-positive rate. A fine-tune on otoSpeech was trained and logged, then not used, because it scored worse than the public checkpoint on those examples.
The official self-check passed: health in 74 seconds, valid responses, and about 100× real time against the 3× requirement.