Voice endpointing
- Score
- 65.2
- Wallet spend
- $5.94 of $50
- Model cost
- $26.29
- Agent time
- 4h 7m
- Steps
- 594
- Tokens in / out
- 2.2M / 593k
- Self-checks
- 1 passed of 2 used
- Rank at $50 wallet
- 2nd of 19
Score
65.2raw metric 0.8798
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.5683
- recall
- 0.8798
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1340
- fn
- 183
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelServing goes serve.sh → server.py /predict → CausalTurnTakingPipeline.predict_conversation in model_pipeline.py. The pip
Serving goes serve.sh → server.py /predict → CausalTurnTakingPipeline.predict_conversation in model_pipeline.py. The pipeline runs learned networks: a CPC encoder with maai VAD/VAP/BC heads, Ekstedt VAP, Mimi with maai heads, and SmartTurn v3.2, v3.1 and v2. It loads them from the checkpoints listed in architecture.json, and the tensor inventories show real weights. On top of these sits a stacked head, loaded from checkpoints/ensemble_head.pt: LR weights, LightGBM boosters rebuilt with lgb.Booster from model strings, LowRankWindowModel state_dicts and EmbProbe vectors. train.py trains that head with real optimization (sklearn LR, LightGBM, AdamW over 150 epochs). logs/train.log records the 8-fold, 3-fold and full fits and a 21 MB checkpoint, which was copied into the output at trajectory#513. The ensemble_head.pt inventory says inspection was unavailable because the pickle contains numpy objects. That limits what the inventory shows but is not a violation, and the loading code and training log support real model use.
✓causal frame probabilitiesOutput is per 20ms frame at 50Hz. Encoder convolutions are causal or left-padded: the maai 50Hz heads use kernel 2 / str
Output is per 20ms frame at 50Hz. Encoder convolutions are causal or left-padded: the maai 50Hz heads use kernel 2 / stride 2 and the 20Hz heads kernel 5 / stride 5, both padding 0, so each output ends before its frame boundary. Ekstedt uses CConv1d left padding. Mimi uses use_causal_conv=True. The GRU carries its hidden state forward, and ALiBi attention is banded and masked so no query sees a later key. SmartTurn windows end at the current frame end, and segmented SmartTurn uses only past VAD plus a forward fill. Resampling is causal (average pooling or decimation). In train.py, the features map 20Hz and 12.5Hz outputs only to completed frames: k20=((i+1)*2)//5-1 and k12=(i+1)//4-1. Rolling statistics are causal, and the silence timers use maximum.accumulate over past indices. 'Stop+4' indices are clipped with min(idx, ...). causal_smooth uses a trailing window. No future frames are used anywhere.
✓no lookup of evaluation audiomodel_pipeline.py and server.py compute predictions only from the decoded audio, through the neural encoders and the sta
model_pipeline.py and server.py compute predictions only from the decoded audio, through the neural encoders and the stacked head. No hashing, fingerprinting, stored per-conversation outputs or lookup tables keyed on input audio were found. The only trained artifact was fit on the 8 public validation conversations, and a search of the session shows no access to held-out audio.
✓training provenancearchitecture.json declares the training data as /task/public/validation.jsonl and the audio of the 8 shipped ids, with n
architecture.json declares the training data as /task/public/validation.jsonl and the audio of the 8 shipped ids, with no external validation or test splits. train.py loads /task/public/validation.jsonl, or /workspace/val_bundle, which the session at trajectory#130 built from that same public file: 'Prepared Conv' was printed only for ids 20, 21, 22, 94, 95, 122, 124 and 125. logs/train.log shows exactly 8 conversations, with LOCO and group folds over those ids. The pretrained checkpoints have declared public sources: maai-kyoto HF repos, kyutai/mimi, smart-turn, whisper-tiny, and the CPC checkpoint from dl.fbaipublicfiles. Their downloads are logged in logs/download_models.log and in the trajectory (git clone at trajectory#58-80; hf_hub_download calls). Trajectory searches for turn-benchmark, turnbench, mundo, JSALT, load_dataset and dataset repo types found no use of the prohibited TurnBench dev release, held-out ids or public validation/test splits.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENAI, GEMINI and GOOGLE_API keys, BASE_URL, env dumps and LLM CLI launches found no
Trajectory searches for ANTHROPIC, OPENAI, GEMINI and GOOGLE_API keys, BASE_URL, env dumps and LLM CLI launches found no participant use of harness model credentials. The credentials the session did use were HF_TOKEN for public model downloads and the Spin transfer credentials for moving ensemble_head.pt from the GPU machine (trajectory#513). Neither is a harness model-API credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed3h 52m into the runtook 6m 15s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh] Installing vendored wheels from /submission/wheels into /tmp/site-packages ... [serve.s
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve.sh] Installing vendored wheels from /submission/wheels into /tmp/site-packages ... [serve.sh] Starting Causal Turn-Taking Server on 0.0.0.0:8000 (MODEL_NAME=submission) ... Traceback (most recent call last): File "/submission/server.py", line 19, in <module> from model_pipeline import CausalTurnTakingPipeline File "/submission/model_pipeline.py", line 45, in <module> from train import ( File "/submission/train.py", line 32, in <module> import turn_taking ModuleNotFoundError: No module named 'turn_taking'
- 2passed4h 0m into the runtook 4m 46s
startup 72.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 72.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.8471907025294968}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.8494536450025407}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.846418417395762}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.848574278147953}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 1406 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 46.6 s) plus the other 19771 s at conversation 124's pace (617 s in 42.4 s, 14.5x real time). The two conversations alone: 14.6x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Hand-Off Summary: Causal End-of-Turn (EOT) Detection
1. Task Requirements & Critical Rules (/task/CONTRACT.md, /task/validate.py, /task/turn_taking.py)
- NEVER finish a turn without a tool call until the final submission is completely in place in
/workspace/output/(ending a turn without a tool call immediately terminates the session!). - Problem: Given two aligned mono 48 kHz, 24-bit FLAC recordings (
speaker_1_audio,speaker_2_audioas base64 inPOST /predict), return causal per-frame EOT probabilities at fixed hophin[0.02, 0.20]seconds (we useh = 0.02s, 50Hz):{"speaker_1": {"hop_seconds": 0.02, "eot_probability": [...]}, "speaker_2": {"hop_seconds": 0.02, "eot_probability": [...]}}Frame count per speaker must beturn_taking.frame_count(duration_s, h)=math.ceil(duration_s / h - 1e-9). - Causality: Frame
icovers(i*h, (i+1)*h]and is reported at time(i+1)*h. Its valuep_iin[0, 1]can ONLY depend on audio from either channel up to(i+1)*h("lookahead means reporting later, never by backdating"). - Scoring (
turn_taking.score):- Positive windows:
[t - 0.25s, min(t + 1.0s, same_speaker_next_eot)]for eachtingold["eot_positive_events"](381 total across the 8 public conversations). - Negative pause windows:
[start - 0.25s, end]for each[start, end]ingold["eot_negative_spans"](244 total across the 8 public conversations). - Frames inside
gold["eot_excluded"]of that speaker are skipped; frames inside a positive window of that speaker are skipped in negative windows. - A window's statistic is the maximum probability reported inside it.
- Score: highest
recall(T)withFPR(T) <= 0.05pooled across all conversations.
- Positive windows:
- Data Rules:
- Do NOT use validation/test splits of public datasets, and do NOT use any other copy/derivative of TurnBench beyond the 8 shipped conversations in
/task/public/validation.jsonland/task/public/audio/{20,21,22,94,95,122,124,125}/speaker_{1,2}.flac. - Free to use the 8 provided validation conversations however we want, plus pretrained checkpoints.
- Do NOT use validation/test splits of public datasets, and do NOT use any other copy/derivative of TurnBench beyond the 8 shipped conversations in
- Serving & Packaging Requirements (
/workspace/output/):bash /submission/serve.shreceivesHOST=0.0.0.0,PORT=8000,MODEL_NAME=submission.GET /healthmust return HTTP 200 within 300 seconds.- Runs offline on 1x H100 SXM 80GB, 8 CPU cores, 80 GiB RAM, Ubuntu 24.04, CUDA 12.8 runtime, Python 3.12 in
/opt/venv(pip, numpy, pillow, safetensors, httpx preinstalled; PyTorch NOT preinstalled). - Must run at least 3x faster than real time (
<= audio_seconds / 3). - No absolute, escaping, dangling, or cyclic symlinks in
/workspace/output/. - Must include
/workspace/output/architecture.jsonwithbase_model,base_revision,training_evidence_files,inference_evidence_files, plus data download scripts and readable training/download logs. - 4 of 4
frontier selfcheckruns remain available.