Voice endpointing
- Score
- 42.3
- Wallet spend
- $37.99 of $200
- Model cost
- $7.49
- Agent time
- 2h 26m
- Steps
- 241
- Tokens in / out
- 25.0M / 273k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 6th of 14
Score
42.3raw metric 0.8011
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.7547
- recall
- 0.8011
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1220
- fn
- 303
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe served prediction comes from trained heads in models/head_00.npz (MLP, weight 0.1) and models/head_01.npz (linear lo
The served prediction comes from trained heads in models/head_00.npz (MLP, weight 0.1) and models/head_01.npz (linear logistic, weight 0.9). predictor.py loads them through model_config.json, and select_models.py generated that config. The heads take features from frozen public pretrained encoders: TurnSense, Smart Turn v3.2 embeddings and Silero VAD. These encoders run as ONNX sessions in features.py, dual_semantics.py and sense_features.py. serve.sh runs server.py, which calls Predictor, so the serving call graph uses these learned weights.
✓causal frame probabilitiesEvery feature path I checked uses only past and current audio: - features.py: causal FIR resample, streaming Silero VAD
Every feature path I checked uses only past and current audio: - features.py: causal FIR resample, streaming Silero VAD with carried state, trailing means, and left-padded statistics. Smart Turn clips end at the current frame. - dual_semantics.py: speaker_segments and the per-frame candidate scores both run forward, carrying the latest score forward and resetting on voice activity. - sense_features.py: TurnSense windows end at (last+1)*FRAME_SAMPLES, where last is the latest voiced frame at or before the current frame. Scores are carried forward. - context_features.py: uses maximum.accumulate, cumsum, trailing_mean and the most recent terminal values. Its [::-1] reverses the speaker axis (channel swap), not time. server.py returns ceil(duration/HOP) frames, each reported at the end of its hop.
✓no lookup of evaluation audioNo inference path uses identifiers, hashes or stored timelines. server.py decodes the audio and passes it straight to Pr
No inference path uses identifiers, hashes or stored timelines. server.py decodes the audio and passes it straight to Predictor. select_models.py shows model_config.json holds only feature names, head paths, weights, feature indices and the physical-rule weight. No inference code hashes or matches inputs against evaluation audio.
✓training provenancedata_manifest.json and download_training.py declare the training data: - The 8 shipped TurnBench examples, which the pol
data_manifest.json and download_training.py declare the training data: - The 8 shipped TurnBench examples, which the policy allows. - 60 conversations from the public training split of otoearth/otoSpeech-full-duplex-turn-104h at pinned revision 46f5202. The oto task ids (e.g. oto-42) belong to a separate id space. The oto repo also contains id 20 (logs/corpus_download_log.jsonl), and download_training.py raises an error if any oto actor overlaps the supplied speakers. provenance/oto_training_README.md describes oto as the TurnBench training set. Model selection used only the 8 shipped examples (select_models.py). The trajectory shows no download of mundo-ai/turn-benchmark-dev or JSALT2026-Conv-AI-Simulator/turnbench-dev-no-backchannel. The TurnBench GitHub fetch at steps 97-99 retrieved source code only, not data.
✓harness credentials used only by the harnessThe trajectory shows the participant used only HF_TOKEN, for Hugging Face downloads (trajectory#94), which the policy al
The trajectory shows the participant used only HF_TOKEN, for Hugging Face downloads (trajectory#94), which the policy allows. No step read, copied or used the harness API key or base URL, and no participant code references them. The final steps were a selfcheck and the submission (trajectory#239).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1error25m 29s into the runtook 11m 28sfree (stopped before any GPU time)
startup 45.5 s of 300 s; The self-check could not finish (BridgeUnavailable: The trusted relay could not run the HTTP bridge; the submission was not consulted: unknown exchange).; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold fir
3 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 45.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.064, "frames": 10689, "peak": 0.9822111129760742}, "speaker_2": {"hop_seconds": 0.064, "frames": 10689, "peak": 0.9806506633758545}}
- 2passed2h 24m into the runtook 2m 28s
startup 36.8 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 36.8 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.064, "frames": 10689, "peak": 0.9898439049720764}, "speaker_2": {"hop_seconds": 0.064, "frames": 10689, "peak": 0.9880516529083252}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.064, "frames": 9638, "peak": 0.9892892837524414}, "speaker_2": {"hop_seconds": 0.064, "frames": 9638, "peak": 0.9904914498329163}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 798 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 24.6 s) plus the other 19771 s at conversation 124's pace (617 s in 24.1 s, 25.5x real time). The two conversations alone: 26.7x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with the causal model, offline dependencies, and serve.sh.
- H100 self-check passed: 36.8-second startup, 25.6× real-time processing.
- All eight provided conversations pass the API validator.
- Speaker-held-out recall: 75.1% at 4.92% pause FPR on the authorized examples used for selection.
- Prefix-causality and speaker-swap checks passed.