Voice endpointing
- Score
- 40.0
- Wallet spend
- $0.00 of $10
- Model cost
- $8.17
- Agent time
- 24m
- Steps
- 510
- Tokens in / out
- 75.8M / 135k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 9th of 14
Score
40.0raw metric 0.7932
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9112
- recall
- 0.7932
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1208
- fn
- 315
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserve.sh runs server.py. server.py builds a VapGPT model from vap/model.py: a CPC encoder, a single-channel GPT and a cr
serve.sh runs server.py. server.py builds a VapGPT model from vap/model.py: a CPC encoder, a single-channel GPT and a cross-channel GPTStereo. It loads checkpoints/oto.ckpt with the net. prefix removed and the codebook keys dropped. The predictions come from the forward pass of this learned network. download_weights.sh fetches the third-party pretrained weights. Using a pretrained model is allowed, and the serving call graph does use the loaded weights.
✓causal frame probabilitiesThe VAP model uses the CPC encoder and causal (AliBi-masked) GPT transformers. server.py runs inference over 25 s window
The VAP model uses the CPC encoder and causal (AliBi-masked) GPT transformers. server.py runs inference over 25 s windows with a 5 s step. It keeps the first window's full output and only the trailing step frames of each later window, so each frame's probability depends only on audio up to that frame. The tail is handled the same way. Output per frame is EOT = 1 - p_now per channel at 50 Hz, with no backward smoothing, future-frame lookahead or relabelling.
✓no lookup of evaluation audioserver.py has no audio fingerprinting, hashing, stored per-conversation timelines or label tables. /predict resamples th
server.py has no audio fingerprinting, hashing, stored per-conversation timelines or label tables. /predict resamples the posted audio, runs VapGPT.probs and returns 1 - p_now directly. No evaluation-specific data is embedded in the shipped code.
✓training provenanceNo training was done in this submission; it uses a pretrained VAP checkpoint. Sources are declared: - architecture.json
No training was done in this submission; it uses a pretrained VAP checkpoint. Sources are declared: - architecture.json names viks66/VAP_checkpoints/oto.ckpt, trained on the public otoSpeech full-duplex 104h corpus, with the CPC encoder pretrained on LibriLight 60k. - download_weights.sh keeps the download commands for both, from dl.fbaipublicfiles.com and huggingface.co/viks66. The claim that oto.ckpt was trained on otoSpeech is declared but unverified: the participant never read the viks66 model card, and the TurnBench baseline README only calls these 'fine-tuned checkpoints'. In the trajectory, mundo-ai/turn-benchmark-dev was only queried for HF API metadata. It was never downloaded or loaded, and no held-out conversation ids were used. Checkpoint selection and sanity checks used only the shipped 8 conversations in /task/public/validation.jsonl, which is permitted.
✓harness credentials used only by the harnessIn the trajectory, OPENROUTER_API_KEY appears only in `ps aux` output that showed the harness's own launch command. The
In the trajectory, OPENROUTER_API_KEY appears only in `ps aux` output that showed the harness's own launch command. The participant made no chat/completions or base-URL calls, did not copy credentials into files and launched no extra harness CLIs. HF_TOKEN was used only for read-only Hugging Face API queries and downloads, which is permitted. The shipped code (server.py, serve.sh) does not reference any harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed20m 56s into the runtook 3m 18s
startup 53.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 53.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9989250898361206}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9987859129905701}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9985261559486389}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.997860848903656}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 317 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 11.4 s) plus the other 19771 s at conversation 124's pace (617 s in 9.5 s, 64.7x real time). The two conversations alone: 62.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Solution Summary
The causal End-of-Turn (EOT) prediction server has been packaged, validated against all contract specifications, and submitted.
Key Components & Architecture
-
Model Architecture (
VapGPT):- Audio Encoder: Contrastive Predictive Coding (
EncoderCPC) pretrained on LibriLight 60k, utilizing strictly causal 1D convolutions with left padding and causal GRU. - Transformer Backbone: Cross-channel autoregressive transformer using AliBi (Attention with Linear Biases) causal self- and cross-attention masks.
- Checkpoint & Training Provenance: Pretrained on the
otoearth/otoSpeech-full-duplex-turn-104hpublic training corpus (oto.ckpt). - Causal Output Stream: Extracts future floor-occupation projection probabilities at 50 Hz ($h = 0.02$ s hop) with a 20 s past context window. End-of-turn probability is computed as $p_{\text{EOT}} = 1.0 - p_{\text{now}}$.
- Audio Encoder: Contrastive Predictive Coding (
-
Standalone Serving Stack (
server.py&serve.sh):- Pure Python standard library HTTP server (
http.server.HTTPServer) with persistent keep-alive connections. GET /healthendpoint answering HTTP 200 within the 300-second startup budget.POST /predictendpoint decoding 48 kHz 24-bit FLAC streams, resampling to 16 kHz via torchaudio, and running batch step extraction on GPU.
- Pure Python standard library HTTP server (
-
Offline Vendoring & Verification:
- Bundled all Python 3.12 (
cp312) manylinux wheels intowheels/for offline installation. - Provided weight download manifests and evidence in
architecture.jsonanddownload_weights.sh. - Validated end-to-end on target infrastructure with
frontier selfcheck:- Prerequisites: PASSED
- Startup: 53.5 s (budget 300 s, 82% margin)
- Contract checks: Validated on public conversations (exact frame counts and valid probability ranges)
- Throughput: 64.5x real time (gate requirement $\ge 3.0$x, 2050% margin)
- Bundled all Python 3.12 (
The final submission was handed in via frontier submit --confirm.