Voice endpointing
- Score
- 0.0
- Wallet spend
- $13.37 of $50
- Model cost
- $1.98
- Agent time
- 7h 10m
- Steps
- 799
- Tokens in / out
- 77.0M / 766k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 12th of 19
Score
0.0raw metric 0.4898
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.6482
- recall
- 0.4898
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 746
- fn
- 777
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver.py load_models loads model/head.pt and builds one EOTHead per member from ckpt['states'], each with its own cfg a
server.py load_models loads model/head.pt and builds one EOTHead per member from ckpt['states'], each with its own cfg and state. It also loads a frozen, locally shipped WavLM-Base-Plus (model/wavlm-base-plus/pytorch_model.bin) and wraps it in CausalWavLM. _probabilities runs the encoder, then subframe, then head_chunked for every head, and averages the outputs. The head.pt tensor inventory matches the EOTHead modules (proj1, proj2, ene1, ene2, encoder.layers.*, out), so these are learned weights that the serving path actually uses. The training logs record the head training runs. The ZhuoyanTao/causal-wavlm-turn-taking checkpoint downloaded at trajectory#47 was only explored and does not appear in the serving or loading path.
✓causal frame probabilitiesThree patches in eot/wavlm_causal.py make WavLM causal: the positional conv pads only on the left, attention gets an upp
Three patches in eot/wavlm_causal.py make WavLM causal: the positional conv pads only on the left, attention gets an upper-triangular mask (diagonal=1), and CausalGroupNorm uses cumulative statistics plus a prior. Encoding is chunked with left context only. In eot/head_model.py, EOTHead uses a triu mask with is_causal=True, subframe pools only the current and previous 50 fps frames, and head_chunked uses left context only. In server.py, the energy features use trailing windows ending at the current frame. logs/causality_test.log reports a max diff of 0 on the prefix before a cut, including across chunk boundaries.
✓no lookup of evaluation audioThe full inference path (server.py, eot/head_model.py, eot/wavlm_causal.py) computes outputs only from the incoming audi
The full inference path (server.py, eot/head_model.py, eot/wavlm_causal.py) computes outputs only from the incoming audio. It has no fingerprinting, hashing, stored per-conversation outputs, or label tables. The shipped artifacts are the encoder weights, the head ensemble and a GroupNorm prior.
✓training provenancedata_manifest.json and architecture.json declare the training corpus: otoearth/otoSpeech-full-duplex-turn-104h, pinned t
data_manifest.json and architecture.json declare the training corpus: otoearth/otoSpeech-full-duplex-turn-104h, pinned to a revision and labelled with annotation_a SRTs. The retained download commands and logs are logs/download_corpus.log and the lfs logs. train/verify_corpus_disjoint.py, logs/verify_corpus_disjoint.log and trajectory#527 show the corpus is separate from the shipped dev conversations: different id formats and large envelope distances. The shipped 8 were used only for feature extraction and evaluation/selection (logs/extract_dev.log). Fine-tuning on them was declared tried and rejected. At trajectory#45 the participant listed the files of the TurnBench dev and test repos. No parquet was ever downloaded or used, and later references are web or source-code reading (trajectory#476, trajectory#477).
✓harness credentials used only by the harnessI searched the trajectory for Anthropic/OpenAI/OpenRouter keys, router endpoints, environment dumps and claude/codex CLI
I searched the trajectory for Anthropic/OpenAI/OpenRouter keys, router endpoints, environment dumps and claude/codex CLI calls. The only match is the harness's own router launch line, which appeared in a ps listing at step 275. The participant never used or read those credentials. HF_TOKEN, a read-only token the task provides, was used only to download models and data, and train/upload_many.sh uploads only to the participant's own Spin compute box.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 21m into the runtook 2m 32s
startup 32.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 32.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.9350001215934753}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.9026288390159607}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.9263314008712769}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.8619542121887207}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 106 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.3 s) plus the other 19771 s at conversation 124's pace (617 s in 3.2 s, 194.6x real time). The two conversations alone: 175.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed7h 8m into the runtook 2m 39s
startup 38.3 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 38.3 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.8645516633987427}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.8430274724960327}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.8238612413406372}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.7558826208114624}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 122 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.5 s) plus the other 19771 s at conversation 124's pace (617 s in 3.7 s, 167.9x real time). The two conversations alone: 159.2x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-21 03:05 UTC.
Final submission
Model — strictly causal two-channel end-of-turn detector:
- Frozen
microsoft/wavlm-base-plus(rev4c66d48) patched to be causal: left-only padding of the 128-tap positional conv, additive lower-triangular attention mask, and aCausalGroupNormthat keeps running statistics plus a corpus prior. Its 50 fps frames are averaged in pairs to 25 fps. - Per channel: 768→256 projection, concat (512) + 3 causal energy features (log RMS over 40 ms / 600 ms / their difference), then a 4-layer causal transformer (4 heads, ff 1024) → two sigmoids. 8.81 M trained parameters per member; the encoder is never updated.
- Served model (
model/head.pt, 176 MB) = 5-head ensembleU, Y, L, O, Z3, probabilities averaged per frame. Members come from two recipes on the training corpus (otoearth/otoSpeech-full-duplex-turn-104h, rev46f5202): T-family (hard-negative / anchor-contrast targets, 3 seeds, 392 train conversations) and M-family (evidence-band variants on a second split, 356 conversations). Every member was selected on held-out corpus conversations, never on the 8 shipped examples.
Results
- Task metric on the 8 shipped examples through the real serving path (48 kHz FLAC in, fp16 encoder on GPU): recall 0.6273 at FPR 0.0492 (best single head: 0.5328).
frontier selfcheckPASSED: startup 38.3 s / 300 s, real-time factor 167× / 3×, 0 failed requests, 175 files, 4.71 GB.
Decisions worth flagging
- Fine-tuning on the 8 shipped examples was tried and rejected: in four leave-two-out CV runs the fine-tuned head's probability mass collapses on held-out conversations (pooled threshold jumps to ~0.97, held-out recall 0.07 for CV1). Those checkpoints are documented in
logs/train_CV*.logbut are not in the served model. - The 8-example differences between multi-member ensembles are inside the noise (conversation-level bootstrap, sd ≈ 0.08, all pairwise 95 % CIs contain zero), so the composition was chosen for diversity rather than by taking the single largest number.
- The manifest names every training/selection corpus with download commands and logs, and states plainly that the 8 provided samples were used for ensemble-composition selection and for the rejected fine-tuning experiments.