Voice endpointing
- Score
- 4.3
- Wallet spend
- $5.76 of $200
- Model cost
- $1.22
- Agent time
- 3h 2m
- Steps
- 204
- Tokens in / out
- 15.4M / 66k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 10th of 19
Score
4.3raw metric 0.6697
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9388
- recall
- 0.6697
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1020
- fn
- 503
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe submission serves a learned model. serve.sh starts serve.py, which loads the pretrained DualTurn model (a Mimi encod
The submission serves a learned model. serve.sh starts serve.py, which loads the pretrained DualTurn model (a Mimi encoder feeding a Qwen2.5-0.5B causal decoder with sigmoid heads) from /submission/model through model/modeling_dualturn.py. That file's from_pretrained reads the shipped model/model.safetensors and model/mimi/model.safetensors. On top of this sits a 12-feature logistic-regression head (model/lr_W.npy, lr_b.npy, lr_mu.npy, lr_sd.npy). The session shows where these came from: the base model and Mimi were downloaded from HF with snapshot_download (trajectory#146, trajectory#147). The LR parameters were trained by train_lr.py and copied into the model directory (trajectory#149). The final check confirmed the LR weights have shape (12,) (trajectory#199).
✓causal frame probabilitiesEach frame's probability depends only on audio up to that frame. modeling_dualturn.py resamples 48 kHz to 24 kHz with to
Each frame's probability depends only on audio up to that frame. modeling_dualturn.py resamples 48 kHz to 24 kHz with torchaudio.functional.resample, a tolerated small lookahead. It then runs the Mimi encoder, which per model/mimi/config.json uses causal convolutions and sliding-window causal attention, followed by the Qwen2 causal decoder over inputs_embeds. serve.py then applies a per-frame logistic head to the same frame's DualTurn outputs, with no smoothing, rolling window, future context or bidirectional pass. A rolling-feature variant was tried in trajectory#197 but not shipped, and it was causal anyway. Output hop is 0.08 s.
✓no lookup of evaluation audioserve.py computes model outputs from the request audio and applies fixed LR parameters. There is no fingerprinting, hash
serve.py computes model outputs from the request audio and applies fixed LR parameters. There is no fingerprinting, hashing, conversation-id table, stored labels or any per-conversation lookup. The shipped artifacts are generic model weights plus 12-dimensional LR parameters. Nothing in the trajectory accesses held-out audio.
✓training provenanceTraining data provenance is declared and acceptable. The base DualTurn model is used as published. architecture.json and
Training data provenance is declared and acceptable. The base DualTurn model is used as published. architecture.json and the HF card/config (trajectory#25, trajectory#28) declare it was trained on the dualturn-switchboard and otospeech turn-taking training data. The only training the participant did was train_lr.py, which reads /task/public/validation.jsonl and computes features from the 8 shipped public conversations (ids 20, 21, 22, 94, 95, 122, 124, 125), which the policy allows (trajectory#195 loops over exactly those ids). Searches of the shipped files and the trajectory found no access to mundo-ai/turn-benchmark-dev, turnbench-dev-no-backchannel, held-out ids, load_dataset calls, or validation/test splits of other datasets. Every HF download in the session was a model (trajectory#146, trajectory#147). cv_lr.py, cv_mlp.py and analyze.py are listed in architecture.json but not shipped; per the session they also used only the 8 public conversations.
✓harness credentials used only by the harnessThe trajectory shows no use of Anthropic, OpenAI or OpenRouter keys, no harness base URLs and no claude or codex CLI lau
The trajectory shows no use of Anthropic, OpenAI or OpenRouter keys, no harness base URLs and no claude or codex CLI launches. Searches for ANTHROPIC, OPENROUTER and claude returned nothing in the trajectory. The credentials the participant did handle were Spin compute and Spin transfer/R2 storage credentials, used to move public audio to the rented GPU (trajectory#68, trajectory#70). Step 13 only lists OpenRouter models through Spin. Both are permitted under the policy.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed2h 32m into the runtook 15m 8s
startup 83.9 s of 300 s; failed POST /predict (conversation 122): RuntimeError: connection: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: R
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 83.9 s
- ✗POST /predict (conversation 122) · RuntimeError: connection: failed 3 attempts: RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response | RuntimeError: Submission HTTP request failed: RemoteDisconnected: Remote end closed connection without response
- 2passed2h 48m into the runtook 3m 49s
startup 84.1 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.1 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9826087951660156}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9822694659233093}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9789465665817261}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9835789203643799}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 121 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.9 s) plus the other 19771 s at conversation 124's pace (617 s in 3.6 s, 170.5x real time). The two conversations alone: 152.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.