Voice endpointing
- Score
- 42.3
- Wallet spend
- $8.91 of $50
- Model cost
- $0.82
- Agent time
- 1h 35m
- Steps
- 351
- Tokens in / out
- 35.0M / 308k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 7th of 19
Score
42.3raw metric 0.8011
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.5664
- recall
- 0.8011
- fpr
- 0.0476
- positives
- 1523
- pauses
- 819
- tp
- 1220
- fn
- 303
- fp
- 39
- tn
- 780
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelsrc/model.py defines a TurnModel trained from scratch. It has a causal-conv front end, 6 causal attention blocks and EOT
src/model.py defines a TurnModel trained from scratch. It has a causal-conv front end, 6 causal attention blocks and EOT/VAD heads. src/train.py trains it with BCE losses and saves state dicts with their args. logs/run1.log, run2.log and run3.log show the loss going down step by step and the evaluations. The weight inventories for weights.pt, weights_b.pt and weights_c.pt contain model.front/inp/blocks tensors, which match the TurnModel shapes. src/server.py loads all three checkpoints into TurnModel, runs infer_streams on each and averages the outputs. So the learned weights are on the serving path. A causal post-rule (silence run × other-speaker VAD) is applied on top of those outputs.
✓causal frame probabilitiesThe convolutions are left-padded (k−1) and attention uses a causal mask. CMVN is a running causal average with replicate
The convolutions are left-padded (k−1) and attention uses a causal mask. CMVN is a running causal average with replicate left padding. src/mel.py uses STFT with center=False, so output frame b depends only on audio up to about 40b+25 ms, before the frame ends. In src/common.py, window stitching takes the second half of each sliding window and tail frames copy the last value forward. Both are causal. The polyphase Kaiser resampler in src/serve_impl.py has a half-width of about 0.6 ms, within the 5 ms allowance. The post-processing in src/server.py (silence_run, VAD gating) uses only past and current frames.
✓no lookup of evaluation audiosrc/server.py and src/common.py compute predictions only from the decoded request audio through the model and causal pos
src/server.py and src/common.py compute predictions only from the decoded request audio through the model and causal post-processing. I found no audio fingerprints, hash tables, stored timelines or per-conversation label lookups in the serving path. src/oracle.py and the src/post_sweep*.py scripts are offline analysis tools that serving never calls.
✓training provenancedata_manifest.json and src/prep_lfs.py declare otoSpeech-full-duplex-turn-104h (otoearth, public TurnBench training set)
data_manifest.json and src/prep_lfs.py declare otoSpeech-full-duplex-turn-104h (otoearth, public TurnBench training set) as the training corpus, plus the 8 shipped provided conversations. src/make_mix.py and src/add_vad.py fine-tune on 6 of them and hold out 20 and 21 for selection. src/prep_dev.py and logs/prep_dev.log read only /task/public/validation.jsonl, ids 20, 21, 22, 94, 95, 122, 124 and 125. In the trajectory, the SesameAILabs/turnbench repo was cloned to /tmp/turnbench (trajectory#15), but only its docs and gold.py were read (trajectory#24). There is no evidence that dev splits, baseline dev predictions, mundo-ai datasets, turnbench-dev-no-backchannel or held-out ids were used. snapshot_download targeted only otoSpeech (trajectory#18 and the steps 24–42 segment).
✓harness credentials used only by the harnessTrajectory searches found no use of ANTHROPIC or harness API keys, no `claude -p` calls and no extra agent CLI sessions.
Trajectory searches found no use of ANTHROPIC or harness API keys, no `claude -p` calls and no extra agent CLI sessions. The only SPIN_API_KEY mention is in Spin documentation output (steps 1–12 segment). HF_TOKEN was used only for otoSpeech and whoami queries (trajectory#18, trajectory#19), and the policy allows that.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 21m into the runtook 4m 8s
startup 23.2 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 23.2 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 1.0}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 1.0}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 1.0}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 1.0}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 252 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 8.7 s) plus the other 19771 s at conversation 124's pace (617 s in 7.6 s, 81.3x real time). The two conversations alone: 79.8x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed1h 30m into the runtook 4m 41s
startup 22.2 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 22.2 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 1.0}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 1.0}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 1.0}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 1.0}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 368 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 12.2 s) plus the other 19771 s at conversation 124's pace (617 s in 11.1 s, 55.6x real time). The two conversations alone: 55.8x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submission is final.
What was submitted (/workspace/output/, 47 files, 349.5 MB):
serve.sh— installs vendored unmodified wheels fromwheels/into/tmp/pylibs(--no-index), then runs the server unprivileged. Verified startup 22 s (limit 300) and 55.6× realtime end-to-end (limit 3×).src/server.py—POST /predictdecodes both FLAC streams, runs three causal checkpoints (60 s windows, 30 s stride, batch 8), averages their per-frame EOT and VAD outputs, then applies a strictly causal "floor-taking" post-process: silence run of the speaker × other speaker's VAD, givinghop_seconds = 0.04and exactlyceil(duration/h)frames per speaker.weights.pt/weights_b.pt/weights_c.pt— three 11.84 M-param causal two-channel transformer checkpoints (log-mel → causal CMVN → strided causal convs → 6-layer RoPE transformer with per-speaker EOT and VAD heads).architecture.json,data_manifest.json,logs/— provenance: from-scratch training onotoearth/otoSpeech-full-duplex-turn-104h(pinned revision, download method and gold-derivation command recorded), fine-tuned on 6 of the 8 provided conversations with 20/21 held out, all checkpoint/post-processing selection on held-out examples only, and aforbidden_sources_not_usedsection confirming no validation/test split of any public benchmark was touched.
Final validation numbers: validate.py recall 0.664 @ 4.1% FPR on the held-out pass (single best model was 0.6535 @ 4.9%; the EOT head alone was ~0.22, the causal floor-taking rule is what carries the score). frontier selfcheck passed on this exact content: startup 22.2 s of 300, 55.6× realtime, healthy after requests, whole-pass projection 368 s of the 6,818 s budget.