Voice endpointing
- Score
- 63.6
- Wallet spend
- $1.35 of $10
- Model cost
- $8.81
- Agent time
- 1h 54m
- Steps
- 272
- Tokens in / out
- 27.2M / 182k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 3rd of 19
Score
63.6raw metric 0.8746
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.8678
- recall
- 0.8746
- fpr
- 0.0464
- positives
- 1523
- pauses
- 819
- tp
- 1332
- fn
- 191
- fp
- 38
- tn
- 781
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe serving path is serve.sh → server.py → inference.py. It builds VapGPT with VapConfig(load_pretrained=0) and strictly
The serving path is serve.sh → server.py → inference.py. It builds VapGPT with VapConfig(load_pretrained=0) and strictly loads weights/vap_oto_state.pt. The inventory shows a full set of CPC encoder, GPT and stereo tensors, which architecture.json traces to viks66/VAP_checkpoints/oto.ckpt at a pinned revision. It also loads a CausalEOTHead from weights/eot_head.pt. That inventory holds trained tensors (norm, input 256x774, a depthwise temporal conv of kernel 9, mix and output) whose sha matches the one recorded in logs/model_selection.json. Training code (training/train_eot_head.py, training/finetune_supplied.py) and logs (logs/train_eot_head.log, logs/finetune_supplied_final.log) are shipped. The prediction comes from these learned models; it is not a heuristic or lookup table.
✓causal frame probabilitiesEach part of the pipeline uses only current and past audio: - Audio is resampled with resample_poly, an FIR filter withi
Each part of the pipeline uses only current and past audio: - Audio is resampled with resample_poly, an FIR filter within tolerance. - The CPC encoder's symmetric padding gives under 10 ms of local lookahead at 100 Hz. The causal CConv1d downsample, which pads on the left only, keeps every 50 Hz frame within its own frame end. - The LSTM is unidirectional, and GPT attention uses a causal tril mask (vap/modules.py). - The EOT head pads its depthwise conv on the left only (kernel-1). - In inference.py, absolute fixed windows (20 s context + 5 s step) emit only frames not yet covered. - Outputs are delayed by 2 frames (zeros prepended), never moved earlier. Shipped logs/causality_final.json records a causality test. No future-frame access was found.
✓no lookup of evaluation audioserver.py decodes the uploaded channels and calls EOTModel.infer_16k. inference.py runs only neural forward passes over
server.py decodes the uploaded channels and calls EOTModel.infer_16k. inference.py runs only neural forward passes over the audio. There is no hashing, fingerprinting, id lookup, embedded label table or stored per-conversation predictions in the serving path. On error the server falls back to zero predictions, not to looked-up answers.
✓training provenanceThe training data has a declared, traceable source: - data_manifest.json and training/prepare_train.py declare and downl
The training data has a declared, traceable source: - data_manifest.json and training/prepare_train.py declare and download otoearth/otoSpeech-full-duplex-turn-104h at pinned revision 46f52029… (ids 1–420, 416 usable conversations). logs/prepare_train.log and logs/features_104h.log record this. - The head split (logs/eot_head_split.json, 374/42) is internal to that corpus. - The 4-fold OOF selection and the final fine-tune use only the 8 permitted ids, 20, 21, 22, 94, 95, 122, 124 and 125. This is confirmed by logs/model_selection.json ('other_benchmark_conversations_or_labels': false), logs/features_public.log and trajectory#80-area feature extraction with --ids 20 21 22 94 95 122 124 125 (trajectory/steps-72-87). - The base VAP and CPC checkpoints have pinned public sources (architecture.json, assets/checkpoints/cpc/60k_epoch4-d0f474de.pt). In the trajectory, the only contact with TurnBench dev was a metadata-only dataset_info listing and a clone of the turnbench code repo whose file list included splits/dev.txt and predictions-dev.json (trajectory#12). Neither file was read or used for training. No held-out ids are used as TurnBench ids, and no validation or test split was used for training.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENROUTER, BASE_URL, API_KEY, FRONTIER_ROUTER and codex found nothing imprope
Searches of the trajectory for ANTHROPIC, OPENROUTER, BASE_URL, API_KEY, FRONTIER_ROUTER and codex found nothing improper: - The router keys appear only in a `ps` process listing showing the harness's own router and claude processes (trajectory#130). The participant did not read, copy, echo or use them. - The participant's S3 helper (trajectory#126) explicitly removes HF_TOKEN and the SPIN keys from the subprocess environment and uses only scoped Spin session credentials. - No model API calls with harness credentials were made, and no extra claude or codex CLI was launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed49m 31s into the runtook 2m 53s
startup 58.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9989234209060669}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9987813234329224}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9984981417655945}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9978659749031067}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 122 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.0 s) plus the other 19771 s at conversation 124's pace (617 s in 3.7 s, 167.3x real time). The two conversations alone: 169.6x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed1h 47m into the runtook 2m 29s
startup 54.0 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 54.0 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9988479614257812}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9987208247184753}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9988213777542114}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.998399019241333}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 124 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.1 s) plus the other 19771 s at conversation 124's pace (617 s in 3.7 s, 165.3x real time). The two conversations alone: 166.7x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
All final checks are complete, and the validation server is stopped. I’m starting the requested two-step final submission for the unchanged self-checked artifact.