Voice endpointing
- Score
- 0.0
- Wallet spend
- $0.00 of $200
- Model cost
- $4.95
- Agent time
- 1h 21m
- Steps
- 290
- Tokens in / out
- 36.0M / 83k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 8th of 14
Score
0.0raw metric 0.2909
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9838
- recall
- 0.2909
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 443
- fn
- 1080
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver.py loads the shipped upstream DualTurn endpointing model, which is a frozen kyutai/mimi causal encoder followed b
server.py loads the shipped upstream DualTurn endpointing model, which is a frozen kyutai/mimi causal encoder followed by a projection, a unidirectional LSTM and MLP heads. It runs that model on the supplied audio through model(...).eot_probs. The tensor names in models/dualturn-endpointing/model.safetensors match the modules in modeling_dualturn.py (proj, core.lstm, heads, feat_mean/std). The checkpoint was downloaded from HF in trajectory#170, and its size matches the shipped file. Predictions come from learned weights, not hand-written rules.
✓causal frame probabilitiesmodeling_dualturn.py encodes audio with Mimi's causal convolutions (use_causal_conv true in the config), then Mimi's cau
modeling_dualturn.py encodes audio with Mimi's causal convolutions (use_causal_conv true in the config), then Mimi's causal sliding-window encoder transformer, then a causal stride-2 downsample. The result goes into a unidirectional LSTM (bidirectional=False). So frame i only depends on audio up to about (i+1)*80 ms. In server.py, the wav[::2] decimation is pointwise, the speaker-swapped second pass reuses the same causal model, and length fixing either forward-fills the last probability or trims. No future context reaches earlier frames.
✓no lookup of evaluation audioserver.py and modeling_dualturn.py compute outputs only through model inference. Neither file contains audio fingerprint
server.py and modeling_dualturn.py compute outputs only through model inference. Neither file contains audio fingerprints, hashes, stored timelines, conversation-id lists or reference-label tables, and no lookup path was found anywhere in the serving call graph.
✓training provenancedata_manifest.json and architecture.json declare the upstream head as trained on the public Switchboard training split,
data_manifest.json and architecture.json declare the upstream head as trained on the public Switchboard training split, with the model obtained as anyreach-ai/dualturn-endpointing at a pinned revision. That upstream training is unverified because no training log is retained, but the source is declared and public. Only the 8 permitted public conversations were used for zero-shot threshold and speed checks. The trajectory shows no download of TurnBench dev, the held-out ids, or validation/test splits: turn-benchmark-dev appears only in search text and a third-party docstring (trajectory#35), and a Switchboard turn-taking dataset was only inspected, not downloaded (trajectory#104).
✓harness credentials used only by the harnessThe harness credentials appear only in the harness launch line shown by ps aux output (trajectory#99, trajectory#105), w
The harness credentials appear only in the harness launch line shown by ps aux output (trajectory#99, trajectory#105), which counts as permitted listing. The trajectory shows no API calls with harness keys, no copying of credentials and no extra harness sessions. The only spin command was `spin compute offers` (trajectory#132), and download_data.sh uses only HF downloads, which are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed47m 31s into the runtook 4m 15s
startup 90.0 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.0 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9957154393196106}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9939876794815063}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9941611886024475}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9920821189880371}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 119 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.4 s) plus the other 19771 s at conversation 124's pace (617 s in 3.6 s, 172.1x real time). The two conversations alone: 163.4x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed1h 17m into the runtook 3m 33s
startup 73.8 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.8 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9957154393196106}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9939876794815063}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9941611886024475}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9920821189880371}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 97 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 3.5 s) plus the other 19771 s at conversation 124's pace (617 s in 2.9 s, 211.9x real time). The two conversations alone: 201.8x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission has been completed and submitted for grading with frontier submit --confirm.
Summary of Completed Work
-
Architecture & Model Selection:
- Selected the DualTurn Endpointing architecture (
anyreach-ai/dualturn-endpointingbacked by the causal neural audio codeckyutai/mimi). - Operates strictly causally on dual-channel 24 kHz audio with a 12.5 Hz frame rate (0.08 s hop), outputting per-frame end-of-turn probabilities conditioned on both conversation speakers.
- Evaluated on the provided TurnBench validation conversations to optimize recall at a 5% false-positive rate.
- Selected the DualTurn Endpointing architecture (
-
Serving & Interface Compliance:
- Implemented
serve.shandserver.pysupporting:GET /healthresponding with HTTP 200 within 74 seconds (well within the 300 s startup budget).POST /predictreceiving base64-encoded 48 kHz FLAC recordings for both speakers and outputting{"hop_seconds": 0.08, "eot_probability": [...]}streams with exact sample-to-frame alignment ($\lceil \text{duration} / h \rceil$).
- Completely offline with all cp312 manylinux wheels vendored in
/workspace/output/wheels/.
- Implemented
-
Performance & Verification:
- Evaluated under
frontier selfcheckon an NVIDIA H100 GPU:- Startup: 73.8 s (required $\le 300$ s).
- Real-Time Factor (RTF): 211.3x faster than real time (required $\ge 3$x).
- All checks and response validations passed.
- Documentation and metadata created:
architecture.json,data_manifest.json,download_data.sh, andinitialization.log.
- Evaluated under