Voice endpointing
- Score
- 0.0
- Wallet spend
- $4.05 of $10
- Model cost
- $1.45
- Agent time
- 3h 10m
- Steps
- 611
- Tokens in / out
- 56.1M / 493k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 16th of 19
The reviewer found a rule violation; see the Review tab.
Score
0.0
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✗training provenance
The shipped head was fitted on the 8 provided conversations, and otoSpeech is a declared public training corpus. However, the shipped evidence/xet_download_oto.py contains a hard-coded id list, DEV = {20,21,22,38,40,41,42,43,44,68,71,94,95,105,...,172}, holding all 30 held-out TurnBench conversation ids. evidence/README.md and architecture.json describe this list as 'the 38 conversation ids of the TurnBench dev split' (turnbench/splits/dev.txt). The policy states that any of the 30 held-out ids used as TurnBench conversation identifiers in id lists, download commands or manifests and appearing in the submission fails. The list is used to exclude those conversations, but under that rule it still fails. The trajectory also shows the participant reading the TurnBench split lists and querying the mundo-ai/turn-benchmark-dev file tree (trajectory#14, trajectory#23); I found no download of the dev parquet files. The otoSpeech download is retained (download_oto.log); the shipped head itself uses only the provided conversations.
✓learned causal modelserver.py loads weights/eot.pt into EOTModel (eot_model.py). EOTModel runs the frozen VAP network (CPC encoder, stereo G
server.py loads weights/eot.pt into EOTModel (eot_model.py). EOTModel runs the frozen VAP network (CPC encoder, stereo GPT and VAP head) on the received audio, then applies a fitted linear head over nine features. The weights/eot.pt inventory holds the VAP tensors and the head. evidence/fit_final.py, together with loo_small.py's train_fold, fits that head by gradient descent and saves final_head.npz, and the shipped logs record the fit. The trained base model plus a fitted head meets this check.
✓causal frame probabilitieseot_model.py runs the CPC encoder over 25 s chunks in 5 s steps and keeps only the newest 5 s of each chunk. The causal
eot_model.py runs the CPC encoder over 25 s chunks in 5 s steps and keeps only the newest 5 s of each chunk. The causal GPT runs over 1250-frame windows in 250-frame steps, again keeping only the newest frames. Padding is added only after the signal, and the model components run causally. All nine head features look backward only: lags, backward run lengths and a max over the past 0.3 s. The output is then delayed by SHIFT=3 frames (60 ms) to cover the encoder's small convolution lookahead. Resampling uses torchaudio's resample or a 65-tap windowed-sinc FIR, both within the standard-resampler tolerance. I found no smoothing that looks back over earlier frames and no whole-signal processing.
✓no lookup of evaluation audioInference computes VAP scalars from the incoming audio and scores them with a 9-weight linear head. The weights/eot.pt i
Inference computes VAP scalars from the incoming audio and scores them with a 9-weight linear head. The weights/eot.pt inventory contains only VAP network tensors and head parameters. I found no fingerprints, transcripts, event timelines or per-recording lookup tables in the serving path (server.py, eot_model.py).
✓harness credentials used only by the harnessThe harness's router keys appear only in ps output (trajectory#182) and are not used or copied afterwards. HF_TOKEN, whi
The harness's router keys appear only in ps output (trajectory#182) and are not used or copied afterwards. HF_TOKEN, which is a participant download credential and not a harness credential, was written to /workspace/.hftok and used only to download from Hugging Face (trajectory#185-210). I found no participant calls to model APIs through the harness credentials or base URL, and no extra harness CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 42m into the runtook 3m 47s
startup 37.9 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 37.9 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9997474551200867}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9996663928031921}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9996146559715271}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9993846416473389}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 100 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.2 s) plus the other 19771 s at conversation 124's pace (617 s in 3.0 s, 206.4x real time). The two conversations alone: 181.0x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed3h 9m into the runtook 1m 34s
startup 33.4 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 33.4 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9997474551200867}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9996663928031921}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9996146559715271}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9993846416473389}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 88 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 3.7 s) plus the other 19771 s at conversation 124's pace (617 s in 2.6 s, 234.6x real time). The two conversations alone: 205.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. /workspace/output was handed in at 2026-10-20 22:43 UTC.
Final submission
Model: frozen VAP (viks66/VAP_checkpoints @ b9aa0ba1) — CPC encoder + stereo GPT + VAP head — with a 9-feature causal linear head:
score_t = alpha·(−logit p_now_self[t]) + b0 + w·(z_t − mu)/sd
Features: p_now trends over 0.5 s/1 s, other-speaker p_now, p_future, both VADs, silence/speech run lengths, and a 0.3 s max of p_now. All backward-looking, plus a 3-frame (60 ms) report delay that covers the CPC encoder's ~50 ms look-ahead. alpha=1, w=0 reduces exactly to the 1−p_now baseline.
Results (8 provided conversations, recall @ 5 % FPR):
| score | |
|---|---|
| VAP baseline, served | 0.6982 |
| LOO estimate for shipped head | 0.7349 (bootstrap Δ +0.0385, win 0.95) |
| In-sample dev | 0.7454 |
Served through /task/validate.py | 0.7480, 0 failed requests |
Verification: frontier selfcheck passed twice on the grading image (final check: digest df5c82ee…, 85 files; H100 80GB; startup 33.4 s of 300 s; 232.5× real time vs 3× required; healthy after requests). Startup is fully offline — vendored PyPI/PyTorch wheels are unpacked to /tmp, and the CPC checkpoint ships with the submission.
Evidence: architecture.json plus evidence/ (data manifest, training/selection scripts, final_head.npz, validator JSON, and logs for the download, caches, MLP rejections, LOO margin selection, bootstrap, and the rejected otoSpeech-combination run). Re-running the final fit reproduces weights/eot.pt bit-for-bit (identical md5).