Voice endpointing
- Score
- 74.7
- Wallet spend
- $1.80 of $10
- Model cost
- $27.01
- Agent time
- 3h 14m
- Steps
- 210
- Tokens in / out
- 46.9M / 200k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 1st of 14
Score
74.7raw metric 0.9127
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.5983
- recall
- 0.9127
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1390
- fn
- 133
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe served model is learned. serve.sh starts src/server.py, which loads the 9 checkpoints named in weights/ensemble.json
The served model is learned. serve.sh starts src/server.py, which loads the 9 checkpoints named in weights/ensemble.json. Each one is an EOTModel (src/model.py): a 6-layer, d=256 transformer with a causal sliding-window mask (window 256) and RoPE. It reads features from the frozen Mimi encoder (weights/mimi) plus a log-energy feature. The checkpoints were trained by src/train.py: first pretrained from scratch on Seamless Interaction, then only head.* was refit on the 8 shipped conversations. Trajectory#201 copies runs/fth_<m>.pt to eot_<m>.pt, and trajectory#204 installs those files as weights/eot_*.pt. Their config {d:256, layers:6, window:256} matches src/model.py, so the shipped weights are the documented trained heads and the serving call graph actually uses them.
✓causal frame probabilitiesEvery stage only looks backward in time. src/features.py decimates 48 to 24 kHz with a causal FIR filter and runs Mimi i
Every stage only looks backward in time. src/features.py decimates 48 to 24 kHz with a causal FIR filter and runs Mimi in chunks that take only left context. weights/mimi/config.json sets use_causal_conv=true and a sliding-window attention. Log-energy is computed from each frame's own samples. src/model.py masks attention with (rel>=0)&(rel<window). src/server.py averages the ensemble logits, applies a sigmoid, and outputs one probability per 0.08 s frame with no smoothing over future frames. Perturbing either channel at 37.04, 64.00 and 100.08 s changed no prediction before the cut, for both the original and the final ensemble (logs/causality_test.txt; trajectory#203 and #205).
✓no lookup of evaluation audioThe predict path in src/server.py only runs features, then the model, then the ensemble average. It contains no fingerpr
The predict path in src/server.py only runs features, then the model, then the ensemble average. It contains no fingerprinting, hashing, stored-answer table or per-conversation branch. The weights contain only model parameters. No code that matches evaluation audio against stored outputs was found.
✓training provenanceSources are declared in DATA_MANIFEST.md and architecture.json. Stage 1 used Seamless Interaction naturalistic/train onl
Sources are declared in DATA_MANIFEST.md and architecture.json. Stage 1 used Seamless Interaction naturalistic/train only; src/prep_seamless.py filters label=='naturalistic' and split=='train', yielding 2866 pairs. Stage 2 and model selection used the 8 shipped conversations (20, 21, 22, 94, 95, 122, 124, 125) from /task/public/validation.jsonl via src/prep_task.py; logs/task_dl.log shows only those 8 ids were copied. The base model is the frozen kyutai/mimi encoder (src/setup_box.sh). The trajectory read only the README, file list and dataset metadata of the Seamless dataset; it shows no download of Seamless dev or test data, no other TurnBench dev release, and no held-out ids. The Seamless download command is retained in the scripts, but its per-file download logs were not readable, so the public source is recorded as declared and partly unverified.
✓harness credentials used only by the harnessThe trajectory shows no use of harness API keys or base URLs in the participant's code and no copying or echoing of them
The trajectory shows no use of harness API keys or base URLs in the participant's code and no copying or echoing of them. It shows no launch of claude, codex or other harness CLIs; searches for 'claude -p', 'codex' and 'OPENROUTER' returned nothing. The only credential use was HF_TOKEN for Hugging Face reads (trajectory#175, #176; Seamless metadata in steps 17–34), which the policy allows. Compute ran through spin (trajectory#201, #207), which is also allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 8m into the runtook 3m 22s
startup 75.3 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.3 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99997}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99997}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99996}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99997}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 77 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 2.4 s) plus the other 19771 s at conversation 124's pace (617 s in 2.3 s, 263.6x real time). The two conversations alone: 273.4x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed2h 54m into the runtook 4m 50s
startup 56.6 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 56.6 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99994}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99993}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99995}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99996}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 62 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 2.1 s) plus the other 19771 s at conversation 124's pace (617 s in 1.9 s, 327.5x real time). The two conversations alone: 329.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 3passed3h 11m into the runtook 3m 42s
startup 69.8 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.8 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99427}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.99298}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9916}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.99319}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 72 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 2.4 s) plus the other 19771 s at conversation 124's pace (617 s in 2.2 s, 285.4x real time). The two conversations alone: 286.6x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The folder matches what selfcheck #3 tested (139 files, 4.68 GB). Confirming the submission: