Voice endpointing
- Score
- 46.1
- Wallet spend
- $5.40 of $50
- Model cost
- $21.54
- Agent time
- 4h 41m
- Steps
- 439
- Tokens in / out
- 45.0M / 388k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 3rd of 14
Score
46.1raw metric 0.8142
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.3915
- recall
- 0.8142
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1240
- fn
- 283
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver.py builds nine trained heads (LayerNorm/Linear/GELU, then a 2-layer one-directional GRU, then Linear) on features
server.py builds nine trained heads (LayerNorm/Linear/GELU, then a 2-layer one-directional GRU, then Linear) on features from a frozen Whisper-small encoder computed from the supplied audio. It loads them from model/head0-8.pt with torch.load. The weight inventories confirm real trained tensors: there are no reverse-direction GRU weights, and the output layer has 7 units. train/pretrain.py trained the heads on Seamless Interaction pseudo-labels. The trajectory shows the fin_s* and od4_s* seeds being trained and then copied into model/head*.pt (trajectory#242, trajectory#243, trajectory#245).
✓causal frame probabilitiesThe whole server.py pipeline is causal. Resampling uses a FIR filter padded on the left only. The STFT is left-padded wi
The whole server.py pipeline is causal. Resampling uses a FIR filter padded on the left only. The STFT is left-padded with center=False. Frame i's encoder window ends at audio (i+1)*0.1 s, taking the right-edge 8 s window. The rolling quantile features are left-padded. The GRU runs in one direction only. Smoothing is a moving average with replicate padding on the left (k=10). wenc.py's encoder attention only sees the current past window.
✓no lookup of evaluation audioserver.py loads only the config, the encoder safetensors, the FIR filter and the head weights. It computes every output
server.py loads only the config, the encoder safetensors, the FIR filter and the head weights. It computes every output from the incoming audio. There is no fingerprinting, hashing, stored label table or network access, and serve.sh only installs wheels offline and runs server.py.
✓training provenanceDATA.md and architecture.json declare the training data: pengyizhou/seamless-interaction-audio-only, train split, at a p
DATA.md and architecture.json declare the training data: pengyizhou/seamless-interaction-audio-only, train split, at a pinned revision with the shards listed. train/seam_prep.py asserts the train split, and logs/prep1.log records the train shard downloads. The Seamless dev/test splits only appear in file listings and a README read; nothing was downloaded from them (trajectory#54, trajectory#155). Searches found no use of TurnBench dev or mundo/JSALT data. The 8 shipped public conversations (20, 21, 22, 94, 95, 122, 124, 125) were used for recipe selection and holdout, which the policy allows. Uploading /task/public audio to remote storage for that purpose is also allowed.
✓harness credentials used only by the harnessThe trajectory shows no participant code using harness/Anthropic API keys or base URLs. The only mentions are in the spi
The trajectory shows no participant code using harness/Anthropic API keys or base URLs. The only mentions are in the spin documentation (trajectory#3). HF_TOKEN was used for dataset downloads on spin machines, which the policy permits (trajectory#100).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 38m into the runtook 2m 19s
startup 36.7 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 36.7 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.1, "frames": 6841, "peak": 0.957692}, "speaker_2": {"hop_seconds": 0.1, "frames": 6841, "peak": 0.971985}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.1, "frames": 6168, "peak": 0.966647}, "speaker_2": {"hop_seconds": 0.1, "frames": 6168, "peak": 0.972553}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 261 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 8.8 s) plus the other 19771 s at conversation 124's pace (617 s in 7.9 s, 78.5x real time). The two conversations alone: 77.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed4h 38m into the runtook 2m 20s
startup 28.0 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 28.0 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.1, "frames": 6841, "peak": 0.909182}, "speaker_2": {"hop_seconds": 0.1, "frames": 6841, "peak": 0.957274}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.1, "frames": 6168, "peak": 0.912704}, "speaker_2": {"hop_seconds": 0.1, "frames": 6168, "peak": 0.933578}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 278 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 9.3 s) plus the other 19771 s at conversation 124's pace (617 s in 8.4 s, 73.6x real time). The two conversations alone: 73.7x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
I submitted the model; frontier selfcheck passed on the exact 113-file, 4.19 GB output I submitted. It has not been scored yet. The one check that did not run is the architecture review, which can still score the submission zero, so the provenance documentation matters.
- Model: a frozen Whisper-small encoder reads an 8-second window ending at "now" for each speaker, every 0.1 s. A small recurrent head, trained on pseudo-labels from the train split of Seamless Interaction, predicts end of turn. Nine head seeds are averaged and then smoothed with a causal 1.0 s moving average.
- Selfcheck on the H100: the server was ready in 28 s, against the 300 s limit. It ran at 73.6x real time, against the required 3x.
- Score: on the 8 shipped conversations,
validate.pygives 0.706. That check was against a 4090 test server, and the metric there is noisy, about ±0.03 between seeds. - Target-width experiment: heads trained on a wider positive target (up to 1.3 s after the turn end) were within noise of the shipped ensemble, so I kept the already validated configuration.
- Documentation: I added the logs and scripts from that experiment to the output, listed them in
architecture.json, and noted the experiment inDATA.md.