Voice endpointing
- Score
- 19.9
- Wallet spend
- $20.94 of $200
- Model cost
- $13.59
- Agent time
- 6h 19m
- Steps
- 249
- Tokens in / out
- 93.2M / 449k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 7th of 14
Score
19.9raw metric 0.7236
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9526
- recall
- 0.7236
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1102
- fn
- 421
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe served predictor in eot_model.py loads three sets of weights: the DualTurn backbone (Qwen2.5-0.5B) with its heads, t
The served predictor in eot_model.py loads three sets of weights: the DualTurn backbone (Qwen2.5-0.5B) with its heads, the Mimi encoder, and a trained ensemble of 8 window heads (h0-h7 plus mu/sd normalization). The tensor shapes in weights/head/window_head.safetensors and weights/dualturn/dualturn_eot.safetensors match what the code expects. training/train_window_head.py trains the heads with AdamW and BCE on Switchboard and OtoSpeech train features plus the 8 shipped files. server.py calls this predictor on /predict, so the serving path really uses the learned weights.
✓causal frame probabilitiesEvery stage of the pipeline is causal: - The 48→24 kHz FIR decimation pads only on the left. - Mimi uses causal convolut
Every stage of the pipeline is causal: - The 48→24 kHz FIR decimation pads only on the left. - Mimi uses causal convolutions and causal sliding-window attention. - The Qwen2 backbone uses causal attention. - The heads work frame by frame, and mu/sd normalization uses fixed values. The probability for frame i, reported at (i+1)*80 ms, depends only on audio up to that time. When the output is short, the last value is repeated forward; no future samples are used.
✓no lookup of evaluation audioNeither eot_model.py nor server.py contains audio fingerprinting, hashing, stored timelines, transcripts or per-file tab
Neither eot_model.py nor server.py contains audio fingerprinting, hashing, stored timelines, transcripts or per-file tables. The weight files hold only model tensors. Predictions come entirely from running the model on the incoming audio.
✓training provenanceTraining data is declared in data_manifest.json and architecture.json: - DualTurn Switchboard and OtoSpeech turn-taking
Training data is declared in data_manifest.json and architecture.json: - DualTurn Switchboard and OtoSpeech turn-taking datasets at pinned revisions, train shards only. training/fetch_data.sh restricts downloads to the pattern data/train-*, and the trajectory shows the same. - The 8 shipped dev files (ids 20, 21, 22, 94, 95, 122, 124, 125), uploaded from the task files at trajectory#38. These are permitted for training and selection. The only contact with the TurnBench dev-no-backchannel dataset was a metadata curl to the HF API at trajectory#15. The participant then explicitly avoided that dataset (trajectory#16, trajectory#39), and the manifest records it as not downloaded. No held-out ids appear anywhere in the submission, and no val/test shards were used.
✓harness credentials used only by the harnessNeither the code nor the trajectory shows any use of ANTHROPIC_API_KEY or other harness credentials, and no extra claude
Neither the code nor the trajectory shows any use of ANTHROPIC_API_KEY or other harness credentials, and no extra claude/codex CLI sessions were launched. The only credentials used were scoped Spin transfer credentials for moving files (trajectory#37, trajectory#38), which the policy permits.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 26m into the runtook 1m 16sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed2h 44m into the runtook 2m 52s
startup 65.6 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.6 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9999991059303284}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9999966621398926}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9999756217002869}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9999946355819702}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 91 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.2 s) plus the other 19771 s at conversation 124's pace (617 s in 2.7 s, 228.7x real time). The two conversations alone: 189.6x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 3passed6h 7m into the runtook 3m 37s
startup 77.8 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.8 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 1.0}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9999995827674866}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9999997615814209}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9999998807907104}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 101 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 5.0 s) plus the other 19771 s at conversation 124's pace (617 s in 3.0 s, 206.0x real time). The two conversations alone: 162.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 4passed6h 13m into the runtook 6m 15s
startup 77.2 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.2 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 1.0}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9999995827674866}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9999997615814209}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9999998807907104}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 99 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.7 s) plus the other 19771 s at conversation 124's pace (617 s in 2.9 s, 210.4x real time). The two conversations alone: 170.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and graded as it stands in /workspace/output/. Its last selfcheck passed on the same 109 files (6.50 GB), with startup in 77 s against the 300 s limit and about 207× real time against the 3× requirement. The architecture review, which the selfcheck doesn't run, has not reported yet.
What the model is
- It starts from the public DualTurn checkpoint (
anyreach-ai/dualturn-qwen2.5-mimi-0.5B, Apache-2.0). Audio goes through a causal 48→24 kHz decimator and the causal Mimi encoder, then the causal Qwen2.5-0.5B backbone. All of these are frozen. - A small per-frame head reads the final hidden state, DualTurn's 18 head outputs, and the layer-12 hidden state. I trained it with a logistic loss on the maximum logit within each scored window, which matches the metric.
- The head is an 8-member ensemble: 4 heads trained on Switchboard plus OtoSpeech train windows, and 4 on Switchboard train windows. All eight also see the eight shipped conversations at weight 5.
- Every stage is causal. Frame i is reported at (i+1)·0.08 s, and the serving features match the offline training features to fp16 noise.
What the evidence shows (eight shipped conversations, official scorer)
- Zero-shot DualTurn scores 0.549.
- A single Switchboard-only head with no in-domain data scores 0.730 pooled and per-conversation AUC 0.907. Adding layer 12 raises the AUC to 0.921.
- Pooled scores are noisy: they move by 0.02–0.04 between seeds and data mixes. I relied more on the AUC.
Choices I made that you may want to revisit
- In-domain weight. The eight shipped conversations contain only three actor pairs. Plain leave-one-conversation-out suggested a large benefit from in-domain data, but it leaks speakers. Holding out whole actor pairs gives AUC 0.912 at weight 0, 0.917 at weight 5, and 0.912 at weight 20. I kept weight 5 as a small hedge.
- Data mix. Switchboard-only heads scored higher pooled on the eight conversations (0.73 against 0.69 for Switchboard plus OtoSpeech) with equal AUC. I used a blend of both to hedge.
- Rejected approaches. Frame-level BCE training failed badly (about 0.2). Fine-tuning the backbone overfit: per-conversation AUC was about 0.93 but the pooled score was lower.
- Caveat on the held-out score. The eight conversations are the only target-like data, and the final head was trained on them. The validator's in-sample numbers on them (recall 1.0) reflect memorization, not generalization, so the held-out score is unknown.
Provenance
- Training corpora are the train splits of the two DualTurn turn-taking datasets (Switchboard: 1986 sessions; OtoSpeech: 896 sessions). Both are listed as "other" licence on the Hub. Revisions, download commands, and logs are in
data_manifest.json,training/fetch_data.sh, andlogs/. architecture.jsonlists the training and inference evidence files, which all exist.- The base checkpoint's weights are reused from its public release, which is allowed by the task.