Voice endpointing
- Score
- 65.0
- Wallet spend
- $3.90 of $10
- Model cost
- $83.07
- Agent time
- 4h 40m
- Steps
- 137
- Tokens in / out
- 27.5M / 191k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 2nd of 14
Score
65.0raw metric 0.8792
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.9189
- recall
- 0.8792
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1339
- fn
- 184
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelThis is a trained model, backed by training code and logs. eot/train.py loads pretrained microsoft/wavlm-large (WavLMMod
This is a trained model, backed by training code and logs. eot/train.py loads pretrained microsoft/wavlm-large (WavLMModel.from_pretrained). It wraps it in the causal EOTModel defined in eot/causal_wavlm.py: a CausalWavLM encoder plus an EOTHead (layer-weight pooling, self/other projections, causal dwconv, 2 causal attention blocks, 10 outputs). Training uses AdamW on BCE EOT loss plus 0.3·VAD and 0.3·VAP auxiliary losses. The checkpoint chain is run1 → run2 → run4. eot/export_model.py writes each checkpoint's sd['model'] to safetensors. The checkpoint_info files and trajectory step 133 record the step and validation score of each ensemble member. All three safetensors list the same encoder.m.* and head.* tensors (525 keys each). eot/server.py loads every model/*.safetensors into EOTModel at startup and serves them as an ensemble.
✓causal frame probabilitiesFrame i uses only audio up to (i+1)·hop. - **Encoder:** CausalWavLM adds a triu(1) −1e4 mask to the attention/relative-p
Frame i uses only audio up to (i+1)·hop. - **Encoder:** CausalWavLM adds a triu(1) −1e4 mask to the attention/relative-position bias and left-pads the positional conv. The conv frontend's receptive field for frame i is [320i, 320i+400). - **Head:** EOTHead uses a left-padded depthwise conv and SDPA with is_causal=True. - **Inference (eot/infer.py):** inference runs in chunks with left context only. Post-processing is a causal EMA, and the output is shifted by one frame (out[:,1:] = pred[:, :n_out-1]). That one-frame delay covers the 80-sample frontend overhang and the FIR resampler lookahead.
✓no lookup of evaluation audioThe serving path does no lookup. serve.sh starts eot/server.py, which decodes and resamples the audio and then calls pre
The serving path does no lookup. serve.sh starts eot/server.py, which decodes and resamples the audio and then calls predict_streams in eot/infer.py. That function only runs the neural ensemble and the EMA. There are no audio fingerprints, hashes, stored timelines or per-conversation tables in the shipped code or model directory.
✓training provenanceThe training data is public AMI headset audio and annotations, with no forbidden data found. - **Source:** downloaded fr
The training data is public AMI headset audio and annotations, with no forbidden data found. - **Source:** downloaded from the Edinburgh AMI mirror by data/download_ami.sh using data/ami_urls.txt. - **Meeting list:** data/ami_train_meetings.json has 120 meetings. All are seen-training scenario meetings or nonscenario EN/IN meetings. Trajectory step 27 builds the list and explicitly excludes the AMI ASR dev set (ES2011, IS1008, TS3004, IB4001-4005/4010/4011) and test set (EN2002, ES2004, ES2014, IS1009, TS3003). - **Selection and calibration:** checkpoints were chosen and the EMA tuned only on the 8 shipped /task/public conversations. They were copied as val/ in trajectory steps 41-42, and eot/eval_val.py scores against them. - **Exclusions:** searches found no TurnBench dev download beyond the shipped 8 and no held-out ids.
✓harness credentials used only by the harnessThe trajectory shows only allowed credential use. - HF_TOKEN was used only for Hugging Face reads and downloads (traject
The trajectory shows only allowed credential use. - HF_TOKEN was used only for Hugging Face reads and downloads (trajectory step 11; the wavlm-large snapshot_download in data/setup_box.sh), with no uploads. - Compute and transfers went through the spin CLI (e.g. step 133), which is allowed. - validate.py was run with only server-url, data and output arguments (steps 65-66), and no judge key was used. - No ANTHROPIC, OPENAI, OPENROUTER or GEMINI keys or harness base URLs are used, and no claude or codex CLI workers are launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 15m into the runtook 3m 43s
startup 96.9 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 96.9 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9951}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9949}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9951}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9949}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 122 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 4.0 s) plus the other 19771 s at conversation 124's pace (617 s in 3.7 s, 168.1x real time). The two conversations alone: 169.1x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed4h 32m into the runtook 4m 33s
startup 95.8 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 95.8 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9978}, "speaker_2": {"hop_seconds": 0.02, "frames": 34203, "peak": 0.9981}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9979}, "speaker_2": {"hop_seconds": 0.02, "frames": 30840, "peak": 0.9979}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 250 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 8.2 s) plus the other 19771 s at conversation 124's pace (617 s in 7.5 s, 81.9x real time). The two conversations alone: 82.5x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
Step 1000 reaches 0.827 recall at 5% FPR. Let me check the stalled wheel download on the box and, independently, look at the box's network to pip.