Voice endpointing
- Score
- 49.4
- Wallet spend
- $0.98 of $10
- Model cost
- $12.40
- Agent time
- 1h 41m
- Steps
- 241
- Tokens in / out
- 23.5M / 229k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 5th of 14
Score
49.4raw metric 0.8253
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9101
- recall
- 0.8253
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1257
- fn
- 266
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe served model is a learned component. It is a 5-seed LightGBM stacker trained by train/train_stacker.py on 66 causal
The served model is a learned component. It is a 5-seed LightGBM stacker trained by train/train_stacker.py on 66 causal features from the supplied audio, using labels from /task/public/validation.jsonl. It was exported to weights/stacker.json and is evaluated in numpy by eot/gbm.py inside Engine.probs. The export check log shows a difference of about 1e-16. The frozen pretrained parts (Silero VAD, Smart-Turn v3.2 and Whisper-large-v3-turbo ASR) are allowed as encoders, VAD or ASR. logs/train_stacker.log shows the training run, with LOCO recall at 5% FPR of 0.7585.
✓causal frame probabilitiesEvery stage only uses audio up to the frame's end: - to_16k_causal pads on the left only, so each output sample depends
Every stage only uses audio up to the frame's end: - to_16k_causal pads on the left only, so each output sample depends only on past input samples. - Smart-Turn windows end at min((i+1)*1024, n) and pad on the right with zeros. - VAD rolling features look backward only. - The Smart-Turn grid mapping uses only past frames. - Lexical features come from the most recently closed segment. A segment counts as closed only when b+0.3 is at or before the frame end. The Whisper clip ends at b+0.1, which is before that release time. No value is backdated. train/causality_test.py compares truncated runs against the full run.
✓no lookup of evaluation audioeot/server.py base64-decodes the posted FLACs and passes them to Engine.predict, which runs only VAD, Smart-Turn, ASR an
eot/server.py base64-decodes the posted FLACs and passes them to Engine.predict, which runs only VAD, Smart-Turn, ASR and the stacker. Warmup uses synthetic noise. There are no fingerprints, hashes, stored timelines or label tables in the serving code or the weights.
✓training provenanceThe stacker's training data is only the 8 shipped conversations (20, 21, 22, 94, 95, 122, 124, 125). This is declared in
The stacker's training data is only the 8 shipped conversations (20, 21, 22, 94, 95, 122, 124, 125). This is declared in architecture.json and DATA_MANIFEST.md and confirmed by logs/extract_signals.log, logs/train_stacker.log and the signals/*.pkl names. Pretrained sources are public and declared with revisions: openai/whisper-large-v3-turbo, pipecat-ai/smart-turn-v3 and silero-vad 6.2.3 from PyPI. Their downloads are logged in logs/download_models.log. The trajectory has no TurnBench dev release or derivative, no held-out ids and no validation or test splits of public datasets. Some third-party HF datasets were looked at during exploration (dataset_info and README files), but the shipped training does not use them.
✓harness credentials used only by the harnessNo trajectory step reads, uses or copies ANTHROPIC/OPENAI or other harness keys or base URLs, and no extra harness CLI s
No trajectory step reads, uses or copies ANTHROPIC/OPENAI or other harness keys or base URLs, and no extra harness CLI session was started. The LLM experiment used a local Qwen2.5-7B-Instruct on spin GPU compute. Spin storage and compute and HF downloads are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed1h 30m into the runtook 1m 3sfree (stopped before any GPU time)
failed review prerequisites: Initialization metadata must declare a nonempty base_model and base_revision
1 gates
- ✗review prerequisites · Initialization metadata must declare a nonempty base_model and base_revision
- 2passed1h 37m into the runtook 3m 13s
startup 63.8 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 63.8 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.032, "frames": 21377, "peak": 0.9834}, "speaker_2": {"hop_seconds": 0.032, "frames": 21377, "peak": 0.98339}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.032, "frames": 19275, "peak": 0.98335}, "speaker_2": {"hop_seconds": 0.032, "frames": 19275, "peak": 0.98355}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 976 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 29.4 s) plus the other 19771 s at conversation 124's pace (617 s in 29.5 s, 20.9x real time). The two conversations alone: 22.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is handed in, and the session is over. The last frontier selfcheck passed on a real H100: /health answered after 63.8 s (limit 300 s), and speed was about 21x real time (required 3x). I haven't seen a score, because the selfcheck doesn't report one and the architecture review didn't run. The selfcheck's own note says the review can still score the submission zero.
- Model: pretrained Silero VAD, Smart-Turn v3.2 and Whisper-large-v3-turbo are used frozen. A small LightGBM stacker, trained only on the 8 shipped labeled conversations, combines their outputs.
- Expected accuracy: leave-one-conversation-out recall at 5% false-positive rate was 0.76. That is noisy, since it rests on 381 positive and 244 pause windows.
- Causality: a truncation test showed no leakage. Predictions on the first 200 s of a conversation matched the full-file run except for a few frames, where one speaker's value differed by up to 0.08. These are the same frames for both cut points, so they are from batched fp16 transcription differences rather than a causal leak.
- Robustness: silent speakers, a 0.05 s clip, mismatched speaker lengths and 16-bit stereo input all returned valid responses.
- Provenance:
architecture.jsonandDATA_MANIFEST.mdname the base models with revisions. They state that no public benchmark split was used for training or selection. - Not used: I considered
livekit/turn-detectorand skipped it, because its licence limits it to the LiveKit Agents framework.
Everything is in /workspace/output/, and the GPU lease is stopped.