Best Choices for Streaming Responses in LLM Applications: A Front-End Perspective

TL;DR: For most LLM apps, SSE is the simplest and most reliable way to stream tokens from your server to the browser. Use WebSockets when you need true bi-directional realtime (tool progress, multi-party collab, cursor sharing). Keep fetch streaming over HTTP/2/3 for controlled environments or server-component frameworks. Design a structured event protocol, add abort/interrupt, reconnect with resume, and instrument TTFT (time-to-first-token) and tokens/sec. Ship with edge-friendly routes, short-lived auth, and ARIA live updates.

1) What “good streaming UX” actually means

• Fast first paint: TTFT < 300–700 ms feels snappy.

• Steady cadence: avoid bursty token dumps; render in small chunks.

• Interruptible: user can “Stop” instantly (AbortController) and edit prompt.

• Recoverable: reconnect on network blips without duplicating output.

• Structured: the stream carries typed events (token, tool, usage, error), not raw text.

• Accessible: screen readers get incremental updates (ARIA live regions).

• Traceable: every stream has correlation IDs and usage counters.

2) Transport options (browser-first view)

Rule of thumb

• Start with SSE for token streaming to the browser.

• Use WebSockets when you truly need client→server push while streaming (tool UI, multi-user cursors).

• Prefer fetch streaming in Next.js/Remix/SvelteKit SSR routes or edge functions where it fits naturally.

• WebTransport is powerful but not yet mainstream for LLM apps.

3) A robust streaming event protocol

Avoid “just send text.” Send framed JSON events with minimal overhead.

Event types

• token: partial text chunk

• delta: structured JSON delta (for tool calls/JSON mode)

• tool_start / tool_result

• message_refusal / safety_violation

• usage: token counts, model, timings

• final: end-of-stream with metadata

• error: typed error

NDJSON framing (works over SSE/WebSocket/fetch)

1{"type":"token","id":"m1","content":"Hello"}
2{"type":"token","id":"m1","content":" world"}
3{"type":"usage","prompt_tokens":342,"completion_tokens":89}
4{"type":"final","id":"m1","finish_reason":"stop"}
5

SSE framing

1event: token
2data: {"id":"m1","content":"Hello"}
3
4event: token
5data: {"id":"m1","content":" world"}
6
7event: final
8data: {"id":"m1","finish_reason":"stop"}
9

Keep payload small, gzip enabled, and include a cursor/offset for resume.

4) Front-end patterns (React examples)

4.1 Streaming with SSE (React + EventSource)

 1import { useEffect, useRef, useState } from "react";
 2
 3export function StreamedAnswer({ requestId, body }: {requestId: string; body: any}) {
 4  const [text, setText] = useState("");
 5  const esRef = useRef<EventSource | null>(null);
 6
 7  useEffect(() => {
 8    const params = new URLSearchParams({ requestId });
 9    const es = new EventSource(`/api/stream?${params}`, { withCredentials: true });
10    esRef.current = es;
11
12    es.addEventListener("token", (ev) => {
13      const msg = JSON.parse((ev as MessageEvent).data);
14      setText((t) => t + (msg.content ?? ""));
15    });
16
17    es.addEventListener("final", () => es.close());
18    es.addEventListener("error", () => es.close());
19
20    return () => es.close();
21  }, [requestId]);
22
23  return (
24    <div aria-live="polite" aria-busy={text.length === 0} className="prose">
25      {text || "Thinking…"}
26    </div>
27  );
28}
29

Notes

• aria-live="polite" announces incremental text for assistive tech.

• Use a “Stop” button calling a /abort endpoint (or keep a client token to cancel; see below).

4.2 Streaming with fetch + ReadableStream (works with HTTP/2/3)

 1async function* streamLines(res: Response) {
 2  const reader = res.body!.getReader();
 3  const decoder = new TextDecoder();
 4  let buf = "";
 5  for (;;) {
 6    const { done, value } = await reader.read();
 7    if (done) break;
 8    buf += decoder.decode(value, { stream: true });
 9    let idx;
10    while ((idx = buf.indexOf("\n")) >= 0) {
11      yield buf.slice(0, idx); // NDJSON line
12      buf = buf.slice(idx + 1);
13    }
14  }
15}
16
17export async function streamAnswer(signal: AbortSignal, payload: any, onToken: (t: string)=>void) {
18  const res = await fetch("/api/stream", {
19    method: "POST",
20    headers: { "Content-Type": "application/json" },
21    body: JSON.stringify(payload),
22    signal
23  });
24  for await (const line of streamLines(res)) {
25    const msg = JSON.parse(line);
26    if (msg.type === "token") onToken(msg.content);
27    if (msg.type === "final") break;
28  }
29}
30

4.3 Immediate Cancel/Interrupt

1const acRef = useRef<AbortController | null>(null);
2function onSend(payload) {
3  acRef.current?.abort(); // cancel previous
4  const ac = new AbortController();
5  acRef.current = ac;
6  streamAnswer(ac.signal, payload, (t) => setText((x) => x + t));
7}
8function onStop() { acRef.current?.abort(); }
9

5) Backpressure, buffering, and rendering

• Batch small tokens: render every 30–60 ms or per ~20–60 characters to avoid reflow storms.

• Use requestIdleCallback (with a fallback) to schedule heavy DOM updates.

• Virtualize long transcripts (e.g., react-virtuoso) to keep DOM light.

• For code blocks, buffer until fenced block closes to avoid flicker.

6) Front-end stack guidance

• React + Next.js: great DX, easy edge routes, supports streaming responses and Server Components; co-locate auth and proxies.

• Remix: nice streaming via loaders and defer, good for progressively rendering pages.

• SvelteKit: minimal overhead, great perf; readable streams supported.

• State: local state for the live answer; server cache (SWR/TanStack Query) for settled results.

• Routing: keep streaming endpoints separate from JSON APIs to tune timeouts and headers.

When to use WebSockets

• Live tool progress (multi-step tool execution).

• Agentic multi-actor UIs (planner/worker logs).

• Multi-user sessions (co-browsing, handoff to human agents).

7) Server routes (Edge/Node examples)

7.1 SSE in a Node/Express route

 1app.get("/api/stream", async (req, res) => {
 2  res.setHeader("Content-Type", "text/event-stream");
 3  res.setHeader("Cache-Control", "no-cache, no-transform");
 4  res.setHeader("Connection", "keep-alive");
 5
 6  const send = (event: string, data: any) => {
 7    res.write(`event: ${event}\n`);
 8    res.write(`data: ${JSON.stringify(data)}\n\n`);
 9  };
10
11  // Example: forward tokens from your model SDK
12  try {
13    for await (const token of model.generateStream({ prompt: req.query.q })) {
14      send("token", { content: token });
15    }
16    send("final", { finish_reason: "stop" });
17    res.end();
18  } catch (e) {
19    send("error", { message: "Stream failed" });
20    res.end();
21  }
22});
23

7.2 WebSocket (ws) with typed events

 1wss.on("connection", (socket) => {
 2  socket.on("message", async (raw) => {
 3    const msg = JSON.parse(raw.toString());
 4    if (msg.type === "new_request") {
 5      for await (const ev of agent.runStream(msg.payload)) {
 6        socket.send(JSON.stringify(ev)); // token/tool_result/usage/final
 7      }
 8    }
 9  });
10});
11

Infra tips

• NGINX/ALB timeouts: raise idle timeouts for long streams (e.g., 60–120s).

• Enable gzip/brotli; add X-Accel-Buffering: no (NGINX) to disable proxy buffering for SSE.

• Prefer keep-alive and HTTP/2 to reduce connection overhead.

8) Product & UX details that matter

• Skeleton + type-on effect: show “Thinking…” immediately; replace with tokens.

• Inline citations: render superscripts as they arrive; clicking opens a side panel.

• Tool feedback: unobtrusive “Calling calendar API…” status chips; collapse into final result.

• Edit & resend: keep the input enabled; allow editing mid-stream by aborting current stream.

• Copy & export: stream into a buffer; final message enables copy/download.

• A11y: role="status" or aria-live="polite"; avoid spamming screen readers (batch updates).

9) Security & privacy

• Never expose model API keys to the browser; front-end talks to your server only.

• Use short-lived session tokens for streaming endpoints.

• CORS: lock origins; prefer same-origin routes behind auth.

• PII controls: redact on server before sending to the model; avoid logging raw prompts.

• Content-Security-Policy: forbid mixed content; pin websocket endpoints.

10) Observability & SLOs

• Metrics per stream: TTFT, tokens/sec, stream duration, bytes sent, abort count, error rate.

• App signals: clarification count, stop ratio, retries.

• Traces: correlate user action → retrieval → model call → stream events.

• Budgets: cap wall-clock (e.g., 30–45s) and token usage per request; surface “truncated” state in UI.

11) Advanced: structured JSON mode & tool results

For tool-augmented agents, stream JSON deltas not just text:

1{"type":"delta","path":"$.thought","value":"Checking your calendar…"}
2{"type":"tool_start","name":"calendar.find","args":{"date":"2025-09-01"}}
3{"type":"tool_result","name":"calendar.find","result":[{"slot":"10:30–11:00"}]}
4{"type":"delta","path":"$.plan[0]","value":"Propose 10:30–11:00"}
5{"type":"final","value":{"answer":"Booked for 10:30","citations":[]}}
6

In React, apply path updates to a local JSON tree, and render partial structure alongside the growing text.

12) Anti-patterns (cost you latency & reliability)

• Sending entire conversation transcript on every turn (use summaries/slots).

• Rendering every single token (coalesce into small batches).

• Building custom binary framing when SSE/NDJSON works fine.

• No abort path—users get stuck waiting.

• Proxy buffering enabled—SSE appears to “not stream.”

13) Quick selection guide

• Simple chat, browser clients, minimal ops → SSE (+ NDJSON events)

• Realtime collab, tool telemetry, agent logs → WebSocket

• SSR/Edge render with progressive responses → fetch streaming (HTTP/2/3)

• Experimental ultra-low latency → WebTransport (only if you know why)

14) Checklist to ship

• Choose transport (SSE/WebSocket/fetch) and define event schema

• Implement abort + “Stop generating”

• Batch tokens; update UI ≤ every 60 ms

• Add TTFT and tokens/sec metrics

• A11y: aria-live + keyboard shortcuts

• Proxy tuned (no buffering, keep-alive, sane timeouts)

• Security: server-side keys, short-lived auth, CORS locked

• Error states + reconnect with offset resume

• Edge deployment (optional): colocate with model/RAG infra

Closing

Great LLM UX isn’t just what the model says—it’s how it arrives. Start with SSE, design a typed event protocol, make it interruptible, and measure the right things. When you need richer realtime semantics, graduate to WebSockets. Keep rendering efficient, accessible, and resilient—and your app will feel fast even when the model is thinking hard.