Voice endpointing
- Score
- 0.0
- Wallet spend
- $1.80 of $10
- Model cost
- $1.34
- Agent time
- 3h 25m
- Steps
- 526
- Tokens in / out
- 51.1M / 643k
- Self-checks
- 3 passed of 4 used
- Rank at $10 wallet
- 16th of 19
Score
0.0raw metric 0.2272
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.4391
- recall
- 0.2272
- fpr
- 0.0476
- positives
- 1523
- pauses
- 819
- tp
- 346
- fn
- 1177
- fp
- 39
- tn
- 780
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe server runs a learned neural network: a plain-PyTorch Whisper-medium encoder with a 5-class turn-taking head (model/
The server runs a learned neural network: a plain-PyTorch Whisper-medium encoder with a 5-class turn-taking head (model/whisper_model.py, strict key loading). Its weights come from model/weights.safetensors: 371 fp16 tensors, with encoder and head keys shown in the weight inventory. The file is the public espnet/Turn_taking_prediction_SWBD checkpoint at a pinned revision, fetched by tools/download_model.sh and converted to fp16 with no other change (logs/export_weights.log, logs/verification.log). server.py loads these weights and calls the model through model/eot.py for every request.
✓causal frame probabilitiesIn model/eot.py, the probability for frame i comes from the mixture window mix[max(0,t-win):t], where t=(i+1)*hop. That
In model/eot.py, the probability for frame i comes from the mixture window mix[max(0,t-win):t], where t=(i+1)*hop. That window is 30 s of past audio ending at the reported frame. The STFT padding stays inside the window, and the log-mel floor is computed per sample. The 201-tap FIR resampler adds about 2 ms of lookahead, which is within tolerance. Speaker attribution uses a past-only energy share: a mean over 8 frames that end 25 frames earlier, excluding the current frame. Stride is 1, so no future values are held or reused. server.py only truncates or pads the output to the frame count.
✓no lookup of evaluation audioThe serving path (server.py, model/eot.py) decodes the request audio, resamples it and runs the model. It holds no store
The serving path (server.py, model/eot.py) decodes the request audio, resamples it and runs the model. It holds no stored timelines, fingerprints, hashes or per-conversation tables. The only files it loads are model/weights.safetensors and model/mel_filters.npz.
✓training provenanceNo weights were updated. The shipped weights are the unchanged upstream ESPnet checkpoint trained on Switchboard; that p
No weights were updated. The shipped weights are the unchanged upstream ESPnet checkpoint trained on Switchboard; that pretraining exposure is upstream and is declared in data_manifest.json and architecture.json. The checkpoint download command is retained in tools/download_model.sh. Hyperparameter and attribution selection (logs/tuning.log) used only the eight shipped conversations, with pooled and leave-one-out scores; TurnBench dev/test are declared not used. In the trajectory, the participant read split lists and filtered the baseline predictions to the shipped 8 only (trajectory#35). Access to otoSpeech and to the HF repos was limited to metadata and listings (trajectory#38, trajectory#230); no dev or held-out audio or labels were downloaded. The remote sweep scored only the shipped 8 (trajectory#241). No held-out ids were used as identifiers.
✓harness credentials used only by the harnessThe trajectory shows only allowed uses: HF_TOKEN for Hugging Face downloads and spin/frontier harness commands (trajecto
The trajectory shows only allowed uses: HF_TOKEN for Hugging Face downloads and spin/frontier harness commands (trajectory#26, trajectory#40). There is no use of harness API keys or base URLs in the participant's own code, no copying of credentials and no launch of other harness CLIs. serve.sh runs offline (HF_HUB_OFFLINE=1) with vendored wheels; all wheel audits are approved.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 20m into the runtook 6m 16s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: installing vendored wheels into /tmp/eot-site ... wheels installed listening on 0.0.0.0:8000 Trace
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: installing vendored wheels into /tmp/eot-site ... wheels installed listening on 0.0.0.0:8000 Traceback (most recent call last): File "/submission/server.py", line 160, in <module> main() File "/submission/server.py", line 148, in main model = load_model(args.weights, args.mel, device=device, dtype=dtype, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/model/whisper_model.py", line 169, in load_model sd = _read_state_dict(ckpt) ^^^^^^^^^^^^^^^^^^^^^^ File "/submission/model/whisper_model.py", line 151, in _read_state_dict from safetensors.torch import load_file File "/tmp/eot-site/safetensors/torch.py", line 5, in <module> from packaging.version import Version ModuleNotFoundError: No module named 'packaging'
- 2passed2h 28m into the runtook 5m 11s
startup 62.0 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 62.0 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.755799}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.927886}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.954491}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.883766}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 2328 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 79.0 s) plus the other 19771 s at conversation 124's pace (617 s in 70.2 s, 8.8x real time). The two conversations alone: 8.7x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 3passed3h 12m into the runtook 4m 56s
startup 67.4 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 67.4 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.628535}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.924495}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.704385}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.794454}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 2314 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 77.3 s) plus the other 19771 s at conversation 124's pace (617 s in 69.8 s, 8.8x real time). The two conversations alone: 8.8x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 4passed3h 19m into the runtook 4m 20s
startup 53.2 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 53.2 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.628535}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.924495}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.704385}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.794454}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 2217 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 75.9 s) plus the other 19771 s at conversation 124's pace (617 s in 66.8 s, 9.2x real time). The two conversations alone: 9.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in at 22:58 UTC — 67 files, 4.55 GB, exactly the tree that passed the final self-check (c0eb91b6a202, digest 8bd3d90f…).
What was served
serve.sh installs the 31 vendored PyPI wheels offline into /tmp/eot-site and starts server.py, which loads the public ESPNet turn-taking checkpoint espnet/Turn_taking_prediction_SWBD (Whisper-medium encoder + 5-class head) re-implemented in plain PyTorch and served in fp16 — verified bit-identical to the openai-whisper reference. No training, fine-tuning or calibration was done: the weights are the released checkpoint, and only inference hyper-parameters were selected, on the 8 shipped conversations.
Per 40 ms frame the model scores a 30 s window that ends exactly at the report time, and the turn-change probability is attributed to the speaker who held the floor 1.0 s earlier (the model's T peak arrives 0.2–0.4 s after the annotated end of turn, so the instantaneous energy share points at the wrong speaker). The lag raises the per-conversation AUC from 0.687 to 0.71 and the score from 0.25 to 0.31.
Measured results
- End-to-end over HTTP through the shipped server, scored with the task's own scorer: score 0.3018 (recall 0.3018 at FPR 0.0492), 381 positive / 244 pause windows; 5852 s of audio in 1617 s = 3.62× real time on the dev 4090.
- Grading harness on the H100: startup 53.2 s of 300 s, real-time factor 9.23× of the required 3×, held-out pass projected at 2217 s of the 6818 s allowed; both public responses accepted with the expected frame counts.
- Selection was cross-checked on two independent runs of the pipeline and required to win on the leave-one-conversation-out estimate in both; several better-AUC alternatives (max-hold, backchannel contrast, running-max normalisation) were rejected because a better ranking is not a better score at a fixed 5 % false-positive budget — that negative evidence is in
logs/tuning.log§4.
Provenance is complete: architecture.json, data_manifest.json (every corpus with its download commands), logs/ (tuning, inference, end-to-end, verification, wheels), and tools/ with the scripts that produced each number.