Channel-level streaming: deliver incremental LLM output to channel adapters (esp. voice TTS)

[pbranchu]

Summary

Channel responses are delivered to channel adapters as a single finished String (via run_agent_loop through the ChannelBridge), so a channel cannot begin acting on the model's output until the entire reply has been generated. A token-streaming path already exists — run_agent_loop_streaming — but it only feeds the SSE/WebSocket kernel API (the web/desktop "live typing" UI); it is not reachable from the channel system. This proposes routing channel dispatch through the streaming path so adapters can consume incremental output.

Why this is needed (for channels, especially voice)

The voice channel is the sharpest case. Today a voice turn is:

[STT] → [LLM generates the ENTIRE reply] → [TTS first chunk] → audio starts

Even with Cartesia WebSocket streaming on the TTS side (sentence-by-sentence audio), the user still waits for the whole LLM reply before the first audio byte, because voice receives the complete String and only then splits it into sentences. With token streaming it becomes:

[STT] → [LLM first sentence] → [TTS] → audio starts   ‖  LLM still writing the rest

• Short replies (greetings, acks, "two plus two") finish generating in ~600–900 ms anyway, so the gain is negligible — which is why a low-TTFT model already lands a voice turn near ~1 s end-to-end without this.
• Longer replies (a paragraph, a multi-sentence explanation) can spend 2–4 s generating; today that's all silence. Streaming would start audio at ~800 ms and flow the rest as it's produced — a 1–3 s perceived-latency win on exactly the turns that feel slowest.

Secondary benefit: it unlocks per-channel first-token telemetry. Metrics like agent_warmup_cold_request_total and accurate *_dispatch_to_first_token histograms need a real "first token" timestamp in the channel path, which doesn't exist while channels only see the finished response.

How it would be done

1. Route channel dispatch through run_agent_loop_streaming instead of run_agent_loop for adapters that opt in.
2. Generalize the ChannelBridge → adapter contract from "return a finished String" to "deliver incremental token deltas, then a completion signal." This is the core change and touches the shared dispatch contract every channel uses, so it must stay backward-compatible.
3. Adapters opt in. Voice consumes deltas → sentence-splits → pushes into the Cartesia continuation as sentences complete. Channels that don't opt in (chat, email, Slack) keep current behavior by buffering deltas to a final String — identical externally.
4. Threading. The streaming events must carry enough context (agent id, conversation/session id, channel) so the adapter routes them correctly; mirror what the SSE path already emits.

Touch points

• crates/openfang-runtime/src/agent_loop.rs — the streaming loop and where channel dispatch selects streaming vs buffered.
• the ChannelBridge / channel dispatch path — new incremental-delivery contract + a buffer-to-final default.
• the channel adapter trait — an optional incremental-delivery method (default buffers to the existing final-String method).
• crates/openfang-channels/src/voice.rs — consume deltas and feed the existing sentence-splitter → Cartesia continuations wiring.

Risk

This changes a contract shared by all channels, so the main risk is regressing chat/email/etc. The mitigation is the backward-compatible default (buffer deltas to a final String), so only opted-in adapters (voice first) change behavior.

Context

Part of the voice-pipeline overhaul (Cartesia WebSocket streaming and the low-TTFT model swap already landed). Filing as the architectural follow-up for incremental LLM delivery into channels. Voice already reaches ~1 s on short turns without this; the value here is on longer replies and on enabling first-token telemetry in channels.