Voice endpointing
- Score
- 3.7
- Wallet spend
- $20.94 of $50
- Model cost
- $7.31
- Agent time
- 3h 49m
- Steps
- 312
- Tokens in / out
- 120.8M / 427k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 7th of 14
Score
3.7raw metric 0.6678
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.8476
- recall
- 0.6678
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1017
- fn
- 506
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver.py loads frozen Mimi from weights/mimi and builds an ensemble from every weights/eot*.pt with torch.load(weights_
server.py loads frozen Mimi from weights/mimi and builds an ensemble from every weights/eot*.pt with torch.load(weights_only=True). Each head is an EOTNet (eotlib/model.py): per-channel mu/sd normalization, Linear(1024->256), GELU, a 2-layer unidirectional GRU, LayerNorm and Linear(256->4). The weights/eot0.pt inventory is a complete state_dict that matches this architecture (mu/sd, inp.0, GRU l0/l1, norm, head; about 1.05M parameters). The train_args.json sidecars link each head to a train_eot.py checkpoint (v4, v7, v8, v9, v12, v13). logs/train_v4.log shows 8000 steps of training saved to /root/ckpt/v4.pt, which matches eot0. The served predictions come from these learned weights.
✓causal frame probabilitieseotlib/audio.py resamples 48k->24k with a Hann-windowed sinc filter whose half-width is about 13 input samples (roughly
eotlib/audio.py resamples 48k->24k with a Hann-windowed sinc filter whose half-width is about 13 input samples (roughly 0.27 ms of lookahead), well within the 5 ms tolerance. mimi_frames uses Mimi's causal conv encoder, chunked with left context only, and a causal encoder transformer, so frame k depends only on audio before (k+1)*80 ms. The EOT head is a unidirectional GRU. server.py zero-pads only past the end of the shorter channel and forward-fills the final value if there are too few frames. Nothing future-dependent is moved earlier.
✓no lookup of evaluation audioThe serving path in server.py and eotlib/audio.py decodes the uploaded FLAC, computes Mimi features and runs the GRU hea
The serving path in server.py and eotlib/audio.py decodes the uploaded FLAC, computes Mimi features and runs the GRU heads. There is no hashing, fingerprinting, stored timeline or per-conversation table, and no access to evaluation labels.
✓training provenancedata_manifest.json declares the DualTurn-Switchboard TRAIN split only. training/extract_dualturn.py filters by splits.js
data_manifest.json declares the DualTurn-Switchboard TRAIN split only. training/extract_dualturn.py filters by splits.json to split=='train' and downloads only the data/train-*.parquet shards. logs/extract_dualturn.log confirms 1986 sessions (220.9 h) from 52 train shards. The train_eot.py monitoring holdout is taken from train sessions. Validation fine-tuning and CV (finetune_val.py, mix_cv.py) use only the shipped 8 conversations from /task/public/validation.jsonl, which is permitted, and the final heads come from train-split-only runs. Mimi is pinned by revision in download_assets.sh. In the trajectory, step 15 is a metadata-only hub API query that listed the turnbench-dev-no-backchannel repo among others; it downloaded no files. Step 16 explicitly excludes the otoSpeech sources. No download of the TurnBench dev repos or the held-out ids was found. The val_cv logs listed in the manifest were not shipped, so that CV is recorded as unverified; this does not affect the provenance of the training data.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN is used only for Hugging Face hub search and downloads (steps 14 and 15), which is permitted
In the trajectory, HF_TOKEN is used only for Hugging Face hub search and downloads (steps 14 and 15), which is permitted. The participant used the provided spin compute CLI for GPU leasing, upload and exec, as the task prompt directs. No ANTHROPIC or OPENAI credentials appear, there are no openrouter model calls or 'claude -p' invocations, and no harness credentials were copied into the shipped code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 9m into the runtook 3m 4s
startup 67.7 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 67.7 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9992719888687134}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9983320236206055}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9995869994163513}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997439980506897}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 96 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 3.3 s) plus the other 19771 s at conversation 124's pace (617 s in 2.9 s, 214.0x real time). The two conversations alone: 210.0x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed2h 12m into the runtook 1m 51s
startup 54.3 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 54.3 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9991329908370972}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9994900226593018}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9989299774169922}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9996539950370789}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 79 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 2.8 s) plus the other 19771 s at conversation 124's pace (617 s in 2.4 s, 259.1x real time). The two conversations alone: 251.6x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 3passed3h 32m into the runtook 2m 57s
startup 68.0 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 68.0 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.997061014175415}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9990190267562866}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9985859990119934}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99925297498703}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 111 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 3.8 s) plus the other 19771 s at conversation 124's pace (617 s in 3.3 s, 184.8x real time). The two conversations alone: 183.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output was graded as it stood at submission (98 files, 4.39 GB), and it passed frontier selfcheck on the grading-style infrastructure.
What was submitted
- Model: a frozen causal Mimi encoder (
kyutai/mimi, CC-BY-4.0) gives 512-d features per channel at 12.5 Hz. A small head (about 1.05M parameters: Linear, 2-layer unidirectional GRU, output) gives per-speaker end-of-turn probabilities. Six such heads are averaged. - Serving:
serve.shinstalls vendored CUDA torch 2.11.0+cu128, transformers 4.57.6 and soundfile offline into/tmp, then startsserver.py. The server answersGET /healthandPOST /predictwith hop 0.08 s. - Causality: verified numerically. Changing audio at or after a frame's end leaves that frame unchanged.
Results
- Selfcheck (final package,
958d57163641): passed. Startup 68 s (limit 300), about 185× real time (required 3×), no errors. - Official
validate.pyon the packaged server (CPU, public validation only): recall 0.646 at 5% FPR, FPR 0.045, no failed requests. This is a diagnostic figure, not the grading run. - Baseline comparison: a single DualTurn-trained head without the label change scores 0.504 on the same 8 conversations.
Data and rules
- Training used only the DualTurn-Switchboard train split, restricted by
splits.json, plus the pretrained Mimi weights. The DualTurn val and test splits were never read. - I excluded the
otoearth/otoSpeech-full-duplex-turn-*datasets andanyreach-ai/dualturn-otospeech-turn-taking. Their file layout matches the shipped examples, so they look like derivatives of this task's conversations. - The 8 shipped validation conversations were not used for training. I used them for evaluation and to choose the label rule. Fine-tuning on them, in several forms, hurt held-out groups in leave-one-group-out CV, so the submission does not include it.
- Provenance is recorded in
architecture.json,data_manifest.json,logs/andweights/eot*.train_args.json. I checked the bundled Mimi weights against the pinned revision's LFS hash.
Decisions to be aware of
- Label rule: DualTurn's "neither" offsets (no handover, no same-speaker resumption) are treated as end-of-turn when the speaker's silence lasts at least 0.5–1.0 s. This was the biggest gain, 0.504 → about 0.575 per head. I chose it on the validation set, which is allowed but means the number is somewhat optimistic.
- Ensembling: averaging heads gave a further gain (0.575 → about 0.63 offline, 0.646 official). The member choice barely changed the ensemble score, so I used all six heads rather than the best subset.
- Noise: with 381 positives, validation differences under about 0.02 are within noise. I treated those as unresolved.
Two things I didn't resolve: the cause of the stored DualTurn codes not matching the stored features after about frame 130, and why the ensemble gains are as large as they are. The held-out score may differ