Voice endpointing
- Score
- 57.4
- Wallet spend
- $5.94 of $200
- Model cost
- $16.52
- Agent time
- 3h 36m
- Steps
- 319
- Tokens in / out
- 33.3M / 274k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
57.4raw metric 0.8529
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.7548
- recall
- 0.8529
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1299
- fn
- 224
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver/server.py load() builds a turnlib.TurnModel for each of the 12 files listed in weights/infer_config.json and weig
server/server.py load() builds a turnlib.TurnModel for each of the 12 files listed in weights/infer_config.json and weights/files.json. It loads each one with load_state_dict and asserts that no keys are missing apart from embed_tokens. The model is a frozen kyutai/mimi causal encoder, then the DualTurn projection, then a Qwen2.5-0.5B causal backbone, then per-speaker EOT heads. The weight inventory for weights/turn_model_0.safetensors has backbone.layers.* Qwen2 tensors (896/4864 dims, F16) that match TurnModel. train/export_weights.py writes these files from runs/*/model.pt with embed_tokens stripped. Those checkpoints come from train/train.py, which initializes from the DualTurn checkpoint. logs/train/runs/F0/log.jsonl records init_missing=[] and a loss curve that falls over 400 steps. So the served output comes from a learned, fine-tuned model.
✓causal frame probabilitiesEvery stage only looks backward in time. train/featlib.py causal_resample uses a one-sided FIR filter, which adds delay
Every stage only looks backward in time. train/featlib.py causal_resample uses a one-sided FIR filter, which adds delay but no lookahead. weights/mimi/config.json sets use_causal_conv true, and Mimi's encoder transformer uses a sliding window. The Qwen2Model in train/turnlib.py takes inputs_embeds with its default causal attention mask. server/server.py run_model runs sliding blocks of 100 frames, each with 200 frames of zero-padded past context, so the output for frame t depends only on frames up to t. Ensemble averaging is done per frame, and there is no smoothing, no post-hoc use of future frames and no TTA (infer_config tta=false). The output is clipped to ceil(dur/0.08) frames.
✓no lookup of evaluation audioserver/server.py predict decodes the request audio, resamples it causally, extracts Mimi features and runs the neural en
server/server.py predict decodes the request audio, resamples it causally, extracts Mimi features and runs the neural ensemble. It does no hashing, fingerprinting, filename or ID matching, and it uses no stored label table. The weights directory holds only model tensors and configs. Nothing in the serving path looks up evaluation audio.
✓training provenancedata/data_manifest.json declares the training corpora: the train splits of dualturn-switchboard and dualturn-otospeech a
data/data_manifest.json declares the training corpora: the train splits of dualturn-switchboard and dualturn-otospeech at pinned revisions, plus the 8 shipped validation conversations and their augmentations. train/prep_public.py reads only data/train-* parquet shards and keeps only sessions marked 'train'; logs/train/oto.log is consistent with this (896 train sessions). train/train.py folds use only IDs 20, 21, 22, 94, 95, 122, 124 and 125. logs/train/ext.log and logs/train/runs/F0/log.jsonl (val_train lists only val_20..val_125 and their aug variants; holdout is empty) confirm the same. No held-out TurnBench IDs appear, and nothing references mundo-ai/turn-benchmark-dev or JSALT2026 turnbench-dev-no-backchannel. The HF dataset searches in the trajectory were browsing only. Models were downloaded with train/download_models.py, pinned to kyutai/mimi and DualTurn revisions. stt-1b was used only in experiments and is not in the final members.
✓harness credentials used only by the harnessThe trajectory shows only an environment listing that filtered out key, token and secret values (trajectory#6) and spin
The trajectory shows only an environment listing that filtered out key, token and secret values (trajectory#6) and spin help/docs mentioning OpenRouter and base URLs (trajectory#5, trajectory#6). No participant code reads or uses ANTHROPIC/OPENAI keys or base URLs, and no other agent CLI was launched. Model downloads used HF only, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 10m into the runtook 3m 41s
startup 67.2 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 67.2 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99957}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99985}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99994}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99974}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 106 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 3.6 s) plus the other 19771 s at conversation 124's pace (617 s in 3.2 s, 193.4x real time). The two conversations alone: 191.0x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed3h 2m into the runtook 4m 19s
startup 88.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 88.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99907}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99936}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99978}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99972}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 214 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 7.1 s) plus the other 19771 s at conversation 124's pace (617 s in 6.4 s, 95.8x real time). The two conversations alone: 95.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 3passed3h 29m into the runtook 5m 39s
startup 116.0 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only th
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 116.0 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99834}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99943}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99977}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99977}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 287 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 9.6 s) plus the other 19771 s at conversation 124's pace (617 s in 8.6 s, 71.4x real time). The two conversations alone: 71.4x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is handed in (01:22:52 UTC) and the session is over. The score isn't known yet: the grader uses the 30 held-out conversations, and my own numbers come only from cross-validation on the 8 shipped examples.
The final package is a 12-model ensemble of fine-tuned copies of the DualTurn base model. Each copy was trained on all 8 shipped conversations plus augmented copies of them. Averaging probabilities across the 12 members gave the best cross-validated results among the ensembling methods I compared.
- Expected quality: ensembles of the same recipes reached about 0.75–0.77 recall at 5% false-positive rate in 4-fold cross-validation, but single runs vary by about ±0.05, so treat that as a rough guide.
- Final self-check: it passed on the exact submitted weights and code. Startup took 116 s of the 300 s allowed. Speed was 71× real time against the 3× required, so the projected 30-conversation pass is well inside the 6,818 s budget.
- Causality and edge cases: the server was tested on short, silent, unequal-length and 16/24/44.1 kHz audio. Streams come back at the right length, and prefixes of a clip give identical earlier frames.
- Server vs training path: the server's outputs match the training-side prediction code (max difference about 0.01, from the fp16-cached features).
- Provenance:
architecture.jsonanddata/data_manifest.jsonname every corpus, base-model revision, augmentation and training run. They point to the download scripts, logs and cross-validation results. Public data is limited to the Switchboard and otoSpeech train splits, and no public validation or test split was used.
Everything is in /workspace/output; the wheels, weights and code are regular files, with no symlinks.