Voice endpointing
- Score
- 48.0
- Wallet spend
- $8.15 of $50
- Model cost
- $1.51
- Agent time
- 2h 10m
- Steps
- 230
- Tokens in / out
- 16.8M / 65k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 5th of 19
Score
48.0raw metric 0.8207
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.8553
- recall
- 0.8207
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1250
- fn
- 273
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe serving path runs a learned model on the request audio. serve.sh execs server.py. server.py's load() calls load_mode
The serving path runs a learned model on the request audio. serve.sh execs server.py. server.py's load() calls load_model(weights.pt, model_class='StereoTransformerModel') from mm-turn-taking/inference/model_loader.py, which loads the state dict into the AudioVAP StereoTransformerModel. The weights.pt inventory is a 23 MB PyTorch state dict whose keys match that architecture: CPC gEncoder convs, a 1024x256 LSTM gAR, a causal downsample, self- and cross-attention transformers, and VAD/VAP heads. The CPC encoder weights come from assets/cpc/60k_epoch4-d0f474de.pt. The model is causal. MultiHeadAttentionAlibi in modules.py applies a lower-triangular mask in both self- and cross-attention. CConv1d pads only on the left, CPCAR is a forward LSTM, and ChannelNorm works per timestep. trajectory#74, #171 and #173 show weights.pt was downloaded from lggvu/audiovap-candor and copied into the output unchanged, and trajectory#201 shows the loader being pointed at the real model class.
✓causal frame probabilitiesEach frame's EOT probability comes only from audio up to that frame. server.py takes channel 0, downsamples 48 kHz to 16
Each frame's EOT probability comes only from audio up to that frame. server.py takes channel 0, downsamples 48 kHz to 16 kHz with resample_poly(1,3), and runs 20 s windows with a 19 s step. It keeps the frames after the 1 s overlap for later windows, zero-pads only the trailing chunk, and maps p_now of the other speaker from VAPDecoder to eot1/eot2. Working through the receptive field of the CPC convs (symmetric padding inside the stride), the causal CConv1d downsample and causal attention, a 50 Hz frame depends on 16 kHz samples up to about 320m+311, which is before the frame end at 320m+320. The only lookahead is the polyphase anti-aliasing filter, a few 48 kHz samples (well under 1 ms), which is inside the 5 ms resampler tolerance. Padding to ceil(duration/0.02) repeats the last value and uses no future audio.
✓no lookup of evaluation audioserver.py has only /health and /predict. /predict decodes the request FLAC and runs the model. There is no fingerprintin
server.py has only /health and /predict. /predict decodes the request FLAC and runs the model. There is no fingerprinting, hashing, table of stored answers, transcript retrieval or external call at serving time. Searches of all evidence for TurnBench repo names and held-out-related strings found nothing. The trajectory shows the provided public validation.jsonl was used only for diagnostics (trajectory#84 onward, trajectory#114 onward), and nothing derived from it is used as a lookup in serving.
✓training provenancearchitecture.json and data_manifest.json declare the base checkpoint as the public HF model lggvu/audiovap-candor. Its m
architecture.json and data_manifest.json declare the base checkpoint as the public HF model lggvu/audiovap-candor. Its model card (trajectory#75) says it was trained on the Candor corpus (fold 0) with the public mm-turn-taking repository, which was cloned in trajectory#77. The checkpoint download is logged in trajectory#74, and download_models.sh keeps the same snapshot_download command. The CPC encoder is the public Libri-light checkpoint that encoder_cpc_components.py fetches from dl.fbaipublicfiles.com. training_log.txt says there was no fine-tuning, and that calibration and diagnostics used only the 8 provided public conversations, which policy allows. The trajectory shows no dataset downloads (no load_dataset or datasets/ calls) and no access to the TurnBench dev release or its derivatives. pipecat-ai/smart-turn-v2 was downloaded for exploration (trajectory#84-113) but does not appear in the shipped serving path.
✓harness credentials used only by the harnessSearches of the shipped files and the trajectory for API_KEY, BASE_URL, ANTHROPIC, OPENROUTER, codex and claude CLI call
Searches of the shipped files and the trajectory for API_KEY, BASE_URL, ANTHROPIC, OPENROUTER, codex and claude CLI calls found no matches. The boto3 credentials used in trajectory#171 and #174 are Spin transfer credentials for the spin-data bucket, used to move artifacts through the Spin storage proxy, which policy allows. No harness credential is read or copied by the shipped code (server.py, serve.sh), and no nested harness CLI was invoked.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 5m into the runtook 3m 27s
startup 69.1 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.1 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.997373640537262}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9964864253997803}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9980730414390564}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9960401654243469}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 133 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 5.3 s) plus the other 19771 s at conversation 124's pace (617 s in 4.0 s, 154.7x real time). The two conversations alone: 140.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.