Voice endpointing
- Score
- 33.6
- Wallet spend
- $0.00 of $10
- Model cost
- $7.75
- Agent time
- 4h 11m
- Steps
- 274
- Tokens in / out
- 91.9M / 389k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 11th of 14
Score
33.6raw metric 0.7708
The raw metric (end-of-turn recall at 5% false positives) is rescaled so that the shipped silence heuristic (recall 0.655 at 5% false positives) scores 0 and perfect recall scores 100, clamped to that range.
Verifier details
- metric
- eot_recall_at_5pct_fpr
- fpr_target
- 0.0500
- threshold
- 0.8580
- recall
- 0.7708
- fpr
- 0.0488
- positives
- 1523
- pauses
- 819
- tp
- 1174
- fn
- 349
- fp
- 40
- tn
- 779
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓learned causal modelserver.py builds an Engine that loads the frozen kyutai/mimi encoder (pinned revision) and the 10 trained head members l
server.py builds an Engine that loads the frozen kyutai/mimi encoder (pinned revision) and the 10 trained head members listed in head/manifest.json, using torch.load(weights_only=True). The heads are unidirectional GRUs and left-padded TCNs defined in eot_model.py. The head/member0.pt inventory (inp.0.weight [64,1040], GRU hidden size 64, output [1,64]) matches the code's GRUHead with d_in 1040. training/train_eval3.py (--final) produces the member files and package_output.py assembles them; logs/fA_gru.log, logs/fC_lse.log and logs/fD_rank.log record the training runs. One inconsistency: architecture.json describes a 4-member ensemble but the manifest lists 10. That is a documentation error, not a violation. The serving prediction is the average of the sigmoids of the trained members.
✓causal frame probabilitiesEvery stage in eot_model.py and server.py is causal. causal_decimate_2x left-pads by TAPS-1 before the stride-2 FIR. The
Every stage in eot_model.py and server.py is causal. causal_decimate_2x left-pads by TAPS-1 before the stride-2 FIR. The AGC gain for frame i is computed before frame i's level is added. The noise floor and run statistics use only past windows. The F0 window ends at the end of the frame. mimi_stream_embed uses streaming conv padding caches and a KV cache. The GRU is unidirectional and the TCN pads only on the left. Frame i is reported at (i+1)*0.08 s, so nothing is backdated. End padding only extends past the end of the file, and outputs are truncated to n frames. training/causal_test.py reports a max diff of 0 between prefix and full-input outputs.
✓no lookup of evaluation audioNeither server.py nor eot_model.py contains fingerprinting, hashing, stored transcripts, timelines or per-file answer ta
Neither server.py nor eot_model.py contains fingerprinting, hashing, stored transcripts, timelines or per-file answer tables. Predictions are computed only from the incoming audio through the encoder and the trained heads.
✓training provenancetraining/common.py reads only /task/public/validation.jsonl, the 8 allowed public conversations. training/extract_feats_
training/common.py reads only /task/public/validation.jsonl, the 8 allowed public conversations. training/extract_feats_agc.py and training/extract_feats_noise.py extract features only from that audio, and train_eval3.py trains on those features using LOCO CV and --final. architecture.json and data/MANIFEST.md declare this corpus. The base model comes from data/download_base_model.py, which pins and sha256-checks the revision. The trajectory shows one AMI train shard downloaded at trajectory#114. It was inspected and dropped because of the domain gap. The diarizers-community/ami check at trajectory#184 looked only at metadata and was dropped at trajectory#185. Neither went into the shipped heads, and both were public train splits. The trajectory has no TurnBench dev, mundo-ai or JSALT access, and no held-out ids were used.
✓harness credentials used only by the harnessNeither the trajectory nor the shipped files contain any ANTHROPIC_* reference. BASE_URL appears only in Spin documentat
Neither the trajectory nor the shipped files contain any ANTHROPIC_* reference. BASE_URL appears only in Spin documentation text. No step launches an extra harness CLI session such as claude -p. HF_TOKEN was used only for Hugging Face downloads and metadata queries (e.g. trajectory#114 and trajectory#184), which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 46m into the runtook 2m 59s
startup 15.1 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 15.1 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9997509717941284}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9997586607933044}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997456669807434}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997581839561462}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 1464 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 46.0 s) plus the other 19771 s at conversation 124's pace (617 s in 44.2 s, 13.9x real time). The two conversations alone: 14.4x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 2passed3h 4m into the runtook 2m 40s
startup 13.4 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 13.4 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9997636079788208}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9997809529304504}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997565150260925}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997693300247192}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 1463 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 49.7 s) plus the other 19771 s at conversation 124's pace (617 s in 44.1 s, 14.0x real time). The two conversations alone: 13.9x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 3passed3h 33m into the runtook 2m 53s
startup 14.4 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 14.4 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9996992945671082}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9997164011001587}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9996959567070007}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997059106826782}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 1719 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 52.8 s) plus the other 19771 s at conversation 124's pace (617 s in 52.0 s, 11.9x real time). The two conversations alone: 12.4x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
- 4passed4h 5m into the runtook 3m 23s
startup 16.9 s of 300 s; 1/1 gates passed; The 3x real-time gate is projected from 2 of 30 requests at concurrency 1: public conversations 122 (the cold first request) and 124 (its pace). Each gets half its audio's duration to answer, the first 300 s more for warm-up; grading bounds only the
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 16.9 s
- ✓POST /predict (conversation 122) · {"conversation": "122", "speaker_1": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9996992945671082}, "speaker_2": {"hop_seconds": 0.08, "frames": 8551, "peak": 0.9997164011001587}}
- ✓POST /predict (conversation 124) · {"conversation": "124", "speaker_1": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9996959567070007}, "speaker_2": {"hop_seconds": 0.08, "frames": 7710, "peak": 0.9997059106826782}}
- ✓real-time factor · Held-out pass projected from 2 of 30 requests at concurrency 1 at 2053 s of server time for 20455 s of audio, 6818 s allowed: the cold first request (conversation 122, 684 s of audio in 72.7 s) plus the other 19771 s at conversation 124's pace (617 s in 61.8 s, 10.0x real time). The two conversations alone: 9.7x real time. Clock: trusted HTTP client inside the serving sandbox; provider transfer/exec latency excluded.
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output (143 files, 625 MB) was handed in with frontier submit --confirm, and the session has ended.
What was submitted
- Model: a frozen Mimi streaming encoder (kyutai/mimi, CC-BY-4.0, pinned revision) on a causally gain-normalised 24 kHz signal. On top sit a small causal head per speaker: 8 GRUs and 2 TCNs (some with pitch features), averaged over 10 members. Every component is causal. Changing audio after time T leaves every output before T unchanged (max difference 0.0).
- Serving:
serve.shunpacks vendored, unmodified wheels (CPU torch, transformers, soundfile) into/tmpand runs a stdlib HTTP server. No GPU is needed. - Training: the 8 shipped validation conversations only, which the task allows. Losses match the scorer: a BCE on each positive window's maximum, a frame-level BCE on pause windows, and zero weight on frames the scorer ignores.
- Provenance:
architecture.json,data/MANIFEST.md, a base-model download script with sha256 check, a verification script for the shipped audio, the training sources, and all logs and CV results.
Results
- Leave-one-conversation-out CV, recall at 5% FPR on pauses: 0.67–0.73 for single runs and about 0.71 for ensemble mixes. The silence-threshold baseline gets 0.425. At 10% FPR the ensembles reach about 0.82–0.84.
- Biggest single gain: dropping the loss weight on non-window frames took recall from 0.31 to 0.73. Frame-level BCE was much worse.
- Selfchecks: all four were used, and the last passed on this exact folder. It starts in about 17 s. The real-time factor is 10–14x against the required 3x, and the projected held-out pass is about 2,000 s of 6,818 s allowed.
Caveats
- Optimistic estimates: every CV number uses the same 8 conversations I used to pick the configuration. The 5% point rests on 12 false positives, so single runs move by about ±0.05. Expect lower recall on the 30 held-out conversations.
- Public data not used: the AMI copies I found on Hugging Face don't provide continuous per-speaker channels, so I didn't use them.
- Architecture review: it did not run during selfcheck, and the contract says it can still score the submission zero. I can't see how it weighs the provenance files.
- Budget: the $10 spin wallet was not spent. Everything ran on the local CPU.