- Published on
UM-SMF: Streaming Meeting Factorization for Low-Latency Agent Speech
- Authors
- Name
- Motions Technologies
UM-SMF: Streaming Meeting Factorization
Closed speech-to-speech APIs such as GPT‑4o Realtime collapse ASR, policy, and TTS into one joint model. That buys low mouth-to-ear delay. It also hides the decision surface.
AetherCrew’s standup room does the opposite on purpose: a factored pipeline so never-do gates, Fin caps, and tool calls stay inspectable.
mic → Chrome STT → meeting turn (LOOK + Grok) → Grok Voice TTS → VRM visemes
Serial composition makes felt delay ≈ Σ stage latencies. Investors and operators ask the same question: can we keep the audit trail and approach realtime feel?
UM-SMF — UmaMeats Streaming Meeting Factorization — is our answer: a hybrid theorem + algorithm for overlapping the factors under an explicit mouth-to-ear budget.
Theorem
A low-latency conversational meeting agent is equivalent to a causal factorization
under the pipeline budget
| Factor | Meaning |
|---|---|
| (e) | Endpoint / utterance boundary |
| (w) | Recognized words (ASR) |
| (a) | Agent acts / spoken beats (policy) |
| (y) | TTS audio |
| (v) | Visemes / face timing |
Realtime APIs approximate the joint in one model. UM-SMF approximates the same joint by running factors streaming and overlapped, so wall-clock delay moves from (\sum \Delta_i) toward (\max \Delta_i) while keeping (P(a\mid w,h,s)) explicit for never-do enforcement.
Algorithm (Phase A)
- Shorter endpointing — flush STT earlier; do not wait for a paragraph of silence.
- Capped LOOK — bound board/PR context so policy TTFT does not dominate.
- Nitro policy routing — Grok 4.6 with latency-aware routing; stream speakable beats via SSE (
meta→beat→done). - TTS prefetch — start Grok Voice on the first speakable sentence chunk; do not wait for the full multi-beat reply.
- Exclusive audio path — one HTML audio element; barge-in aborts in-flight fetch.
Do not ship audio over a chat WebSocket (head-of-line blocking). Stream every layer.
Stack
| Layer | Choice | Cost posture |
|---|---|---|
| Room UI | Ops /crew/room on existing Vercel app | Idle $0 |
| STT | Browser Web Speech | $0 |
| Policy | Grok 4.6 (:nitro) over streamed turn | Pay-per-turn |
| TTS | Grok Voice x-ai/grok-voice-tts-1.0 | ~0.09/standup** |
| Faces | TalkingHead + visemes from audio | No GPU avatar SaaS |
Evidence posture
Tier‑1 (config locks): deterministic cuts from shorter STT flush, LOOK caps, and token budgets (STT flush −600 ms; LOOK cap −2300 ms; max tokens −65.7% in the locked comparison).
Tier‑2 (scenario budgets): modeling overlapped stages estimates a mean ~60.7% reduction in felt mouth-to-ear latency versus the pre–Phase‑A serial path — without buying a joint S2S vendor.
Tier‑3 (live localhost harness): we publish measurement protocols and before/after JSON. Early runs showed OpenRouter TTFT variance dominating wall clock; streaming first-speakable often arrived near full JSON completion, so early TTS barely overlapped. That is a scientific result, not a marketing slide: policy TTFB is the remaining bottleneck, and the factorization makes that bottleneck measurable.
Why this matters
Joint S2S is a product. Factorization is a lab + production posture:
- Never-do lists stay enforceable at (P(a\mid w,h,s)).
- Cost is pay-per-turn, not always-on GPU media.
- Latency work becomes an algorithm with budgets, probes, and falsifiable pass criteria.
Research continues on speculative interim policy, phoneme-timed lips, and tighter endpointing — always with the mouth-to-ear KPI first.