Voice endpointing
- Score
- 50.5
- Wallet spend
- $1.80 of $10
- Model cost
- $27.60
- Agent time
- 56m
- Steps
- 171
- Tokens in / out
- 18.2M / 115k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 3rd of 14
Score
50.5raw metric 0.8293
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9456
- recall
- 0.8293
- fpr
- 0.0476
- positives
- 1523
- pauses
- 819
- tp
- 1263
- fn
- 260
- fp
- 39
- tn
- 780
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThis is a learned model. serve.sh runs server.py. Model.predict_audio computes features from frozen pretrained models: S
This is a learned model. serve.sh runs server.py. Model.predict_audio computes features from frozen pretrained models: Silero VAD (models/silero_vad_16k_sequence.onnx), VAP (models/vap.pt) and a Smart Turn v3.2 embedding (models/smart_v3_embedding.onnx). It then applies LinearPredictor from linear_model.py, which loads models/linear.npz. The weight inventory shows that file holds coef [2,187], intercept [2] and weight [2]. train_ensemble.py fits these two logistic-regression components (C=0.01, MIL positive selection, ensemble weights 0.25 and 0.75) and writes linear.npz. logs/final_training.log records the fit and leave-speaker-pair-out results. vap_features.py loads models/vap.pt with strict=True, and the inventory's tensor names (encoder.encoder.gEncoder...) match the code's VapNet/EncoderCPC structure. Every probability comes from these learned models, not hand-written rules.
✓causal frame probabilitiesEach frame's probability uses only audio up to that frame. audio_features.py resamples with a causal FIR filter (lfilter
Each frame's probability uses only audio up to that frame. audio_features.py resamples with a causal FIR filter (lfilter with firwin(63), then [::3]) and runs Silero in 512-sample blocks, carrying the recurrent state forward. Its trailing statistics, time_since and smart_mel normalisation look only at past samples. vap_features.py runs the CPC encoder and a transformer with causal ALiBi masking over fixed blocks with past context. linear_model.py maps each 32 ms hop to the latest VAP frame already available, using floor((i+1)*HOP/.02)-1, and uses only past lags. semantic_features.py and extract_semantic.py pick candidate points from past VAD only and embed audio[end-128000:end], where end=(i+1)*512, then hold the last value forward. server.py emits one probability per 32 ms hop and gates it on time since past speech. No code path reads future samples.
✓no lookup of evaluation audioNo lookup of evaluation audio was found. server.py computes outputs from the incoming audio through the model pipeline o
No lookup of evaluation audio was found. server.py computes outputs from the incoming audio through the model pipeline only. There is no hashing of the input, no table of stored labels or answers, and no lookup keyed by conversation id. The only sha256 checks are in prepare_training_audio.py, and they verify the eight training recordings against validation.jsonl. Searches of the shipped files found no reference to the held-out conversation ids or the forbidden dev datasets.
✓training provenanceThe training data is properly sourced. training_data.py, prepare_training_audio.py and train_ensemble.py read only /task
The training data is properly sourced. training_data.py, prepare_training_audio.py and train_ensemble.py read only /task/public/validation.jsonl, the eight allowed conversations (20, 21, 22, 94, 95, 122, 124, 125), as also listed in data_manifest.json. logs/final_training.log reports 8 conversations across 3 speaker pairs. Pretrained models are pinned and declared in architecture.json, data_manifest.json, download_models.py and logs/download_models_final.json: Silero VAD at commit 1e261b0, the VAP example checkpoint at commit f39a78b, and pipecat-ai/smart-turn-v3 at revision f766f81. The trajectory confirms these downloads: a git clone at step 9, hf_hub_download at steps 11-12, and train.py written at step 16 reading only validation.jsonl. Searches found no use of mundo-ai/turn-benchmark-dev or the JSALT turnbench-dev-no-backchannel dataset, and no held-out data.
✓harness credentials used only by the harnessNo harness credentials were used. Searches of the trajectory for ANTHROPIC, API_KEY and BASE_URL found nothing. The sess
No harness credentials were used. Searches of the trajectory for ANTHROPIC, API_KEY and BASE_URL found nothing. The session used spin (budget, compute, docs) and an SSH key returned by spin compute for rented GPU access. It also used HF downloads and a frontier selfcheck, all of which are allowed. No shipped file reads or copies harness API keys or base URLs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed17m 13s into the runtook 3m 8s
startup 66.3 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 66.3 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.032, "frames": 21377, "peak": 0.9994298815727234}, "speaker_2": {"hop_seconds": 0.032, "frames": 21377, "peak": 0.9994823932647705}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.032, "frames": 19275, "peak": 0.9996109008789062}, "speaker_2": {"hop_seconds": 0.032, "frames": 19275, "peak": 0.9997260570526123}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 529 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 24.7 s) plus the other 19771 s at conversation 124's pace (617 s in 15.7 s, 39.2x real time). The two conversations alone: 32.2x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed52m 37s into the runtook 4m 40s
startup 78.1 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 78.1 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.032, "frames": 21377, "peak": 0.9995090365409851}, "speaker_2": {"hop_seconds": 0.032, "frames": 21377, "peak": 0.9995827674865723}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.032, "frames": 19275, "peak": 0.9996245503425598}, "speaker_2": {"hop_seconds": 0.032, "frames": 19275, "peak": 0.9997621774673462}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 1804 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 55.2 s) plus the other 19771 s at conversation 124's pace (617 s in 54.6 s, 11.3x real time). The two conversations alone: 11.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
- Offline H100 self-check passed: 78.1-second startup, 11.3× projected real-time throughput.
- Causality checks passed.
- Supplied-example evaluation: 77.4% recall at 4.9% pause false positives.