Voice endpointing
- Score
- 34.1
- Wallet spend
- $5.40 of $10
- Model cost
- $17.33
- Agent time
- 4h 32m
- Steps
- 309
- Tokens in / out
- 91.3M / 388k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 10th of 14
Score
34.1raw metric 0.7728
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.7859
- recall
- 0.7728
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1177
- fn
- 346
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThe served model is a learned network. serve.py loads CausalWavLM from wavlm_causal_base_plus.pt and EOTHead from eot_he
The served model is a learned network. serve.py loads CausalWavLM from wavlm_causal_base_plus.pt and EOTHead from eot_head.pt, then runs inference through predict_conv. convert_wavlm.py builds the encoder by copying weights from microsoft/wavlm-base-plus into a causal re-implementation. Its checks cover bidirectional parity with transformers and bitwise causality when the future part of the input is swapped. The eot_head.pt inventory shows real trained tensors: a subsample conv, a fuse layer, positional embeddings and a 4-layer transformer encoder. train_head.py and logs/stageA.log, logs/stageB.log, logs/stageFT.log and logs/stageFT2.log record the head being trained from scratch on otoSpeech features, then fine-tuned on the 8 shipped examples. The shipped score of 0.7874 matches the stageFT2 log. The trajectory shows that the ZhuoyanTao turn-taking repo was only listed, never downloaded or used.
✓causal frame probabilitiesEvery part of the inference path is causal or looks only backward. causal_wavlm.py uses a triu -inf attention mask, a le
Every part of the inference path is causal or looks only backward. causal_wavlm.py uses a triu -inf attention mask, a left-padded positional conv and a cumulative causal norm. streaming.py and infer.py process audio in chunks that use only left context and carry the norm state forward. head.py left-pads the stride-2 subsample conv and uses a subsequent mask with is_causal=True. The only lookahead is resample_poly, which is a few ms, well inside the 40 ms reporting margin: frame j sees audio up to about j*40+26 ms and is reported at (j+1)*40 ms. serve.py zeroes the first 5 frames and returns ceil(duration/0.04) frames at a 0.04 s hop.
✓no lookup of evaluation audioserve.py's predict() decodes the audio, resamples it, extracts features and runs the model. There is no hashing, fingerp
serve.py's predict() decodes the audio, resamples it, extracts features and runs the model. There is no hashing, fingerprinting, duration matching or table of stored answers anywhere in the serving path or the shipped code.
✓training provenanceThe training data is otoearth/otoSpeech-full-duplex-turn-104h, a public corpus. It is declared in DATA_MANIFEST.md and a
The training data is otoearth/otoSpeech-full-duplex-turn-104h, a public corpus. It is declared in DATA_MANIFEST.md and architecture.json, and its download is retained in scripts/download_train_data.sh and in trajectory steps 25-26. The trajectory and logs/disjoint_check.log show that the conversation ids it shares with the 8 shipped examples have different actors, conversation types and transcripts, so it is not a copy of the task's conversations. Fine-tuning and checkpoint selection used only the 8 shipped examples (ids 20, 21, 22, 94, 95, 122, 124, 125), per logs/stageFT2.log and train_head.py. Searches of the trajectory found no download of mundo-ai/turn-benchmark-dev or -test, no JSALT no-backchannel mirror, and no use of the baseline repo's dev predictions or probs. The participant only read those files and the repo code, and never used held-out ids. WavLM-Base-Plus is pinned to a specific revision.
✓harness credentials used only by the harnessSearches of the trajectory found no ANTHROPIC, OPENAI, OPENROUTER, GEMINI or *_BASE_URL key use and no claude, codex or
Searches of the trajectory found no ANTHROPIC, OPENAI, OPENROUTER, GEMINI or *_BASE_URL key use and no claude, codex or gemini CLI launches. The only credential used was HF_TOKEN, for read-only downloads (curl or huggingface_hub), including on the spin GPU machine, which is allowed. Every 'upload' in the session is a spin compute upload of files to the participant's own compute machine; none is an upload to Hugging Face.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed4h 28m into the runtook 35sfree (stopped before any GPU time)
failed review prerequisites: Initialization metadata must declare a nonempty base_model and base_revision
1 gates
- ✗review prerequisites · Initialization metadata must declare a nonempty base_model and base_revision
- 2passed4h 29m into the runtook 2m 32s
startup 80.5 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 80.5 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.9710310152952941}, "speaker_2": {"hop_seconds": 0.04, "frames": 17102, "peak": 0.975247624559758}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.9771268764710659}, "speaker_2": {"hop_seconds": 0.04, "frames": 15420, "peak": 0.9667990379357103}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 148 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 6.1 s) plus the other 19771 s at conversation 124's pace (617 s in 4.4 s, 139.8x real time). The two conversations alone: 123.2x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests