การสตรีมคำตอบ LLM ไปยังผู้ใช้

By default an LLM call returns the entire response only after generation finishes. For long answers this feels slow.

Streaming sends tokens as they are produced, so the user sees text appear word-by-word — drastically improving perceived responsiveness.

Two latency numbers matter:

• TTFT (time to first token): how long until the first word appears
• Total time: until the full answer is ready

Streaming barely changes total time but makes TTFT the number your users actually feel.

Most LLM APIs stream over Server-Sent Events (SSE): a long-lived HTTP response where each chunk is a small JSON event prefixed with data:.

The stream ends with a special [DONE] marker.

You opt in by setting a flag on the request. The API then returns an iterable stream instead of a single object.

Each chunk carries a delta — the new piece of text. You concatenate deltas to rebuild the full message.

On the frontend you append each delta to the visible message. A simple approach: keep state and re-render on every token.

When the stream closes, finalize: stop the typing indicator, persist the full message, and re-enable the input box.

A stream can fail halfway. Always wrap consumption in try/catch and show whatever partial text you already received rather than discarding it.

Users may want to stop a long answer. Pass an AbortController signal so you can cancel the request and free server resources.

When the model is calling a function, tool-call arguments also arrive as deltas. Buffer them until the call is complete before executing the tool.

Streaming does not change cost — you still pay per token. To count usage, sum the tokens you received, or request a final usage event if the API supports it.

Test your streaming knowledge.

You learned to stream LLM responses: opt in with a stream flag, read incremental delta chunks over SSE, update the UI per token, handle the done signal, manage errors and cancellation, and remember streaming improves perceived latency without changing cost.