0Pricing
Next.js 15 Fullstack (App Router + Server Actions) · Lesson

Streaming AI Responses Token-by-Token

Stream LLM completions to the UI with backpressure-aware readers and abort handling.

Streaming AI Responses Token-by-Token is a free Next.js 15 Fullstack (App Router + Server Actions) lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Next.js 15 Fullstack (App Router + Server Actions) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Stream AI Responses?

When you call a large language model (LLM) like GPT-4 or Claude, the model generates tokens one by one. A typical response may take 5–20 seconds to complete. If you wait for the full response before sending anything to the client, the user stares at a blank screen the entire time.

Streaming solves this: you pipe each token to the browser as it is produced, creating the familiar "typewriter" effect used by ChatGPT, Claude.ai, and Gemini.

  • Perceived latency drops from time-to-full-response to time-to-first-token (often under 300 ms).
  • Users can start reading and even abort early if the answer is already clear.
  • Server memory stays flat — you never buffer the whole response.

In Next.js 15 the primitives you need are ReadableStream, the Web Streams API, and StreamingTextResponse (or a plain Response with a stream body).

How LLM SDKs Expose Streams

Most LLM SDKs return an async iterable or a ReadableStream when you pass stream: true. The Vercel AI SDK unifies these under a single interface.

With the official OpenAI SDK you receive a stream of ChatCompletionChunk objects. Each chunk carries a delta.content string that may be one token, a few characters, or an empty string at the end.

  • OpenAI SDK: openai.chat.completions.create({ stream: true }) returns an AsyncIterable.
  • Vercel AI SDK: streamText() returns a result with result.toDataStreamResponse() ready for Next.js Route Handlers.
  • Anthropic SDK: client.messages.stream() returns an async iterable of MessageStreamEvent.

Regardless of the SDK, the pattern is the same: iterate over chunks, encode each piece, and enqueue it into a ReadableStream that becomes the HTTP response body.

Creating a Streaming Route Handler

A Next.js 15 Route Handler returns a standard Response. You can pass a ReadableStream as the body to stream data to the client. The ReadableStream constructor accepts a start(controller) callback where you push chunks with controller.enqueue() and signal completion with controller.close().

Below is a minimal Route Handler at app/api/chat/route.ts that streams an OpenAI completion token by token.

// app/api/chat/route.ts
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request): Promise<Response> {
  const { prompt } = await req.json() as { prompt: string };

  const completion = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    stream: true,
    messages: [{ role: 'user', content: prompt }],
  });

  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();
      for await (const chunk of completion) {
        const text = chunk.choices[0]?.delta?.content ?? '';
        if (text) {
          controller.enqueue(encoder.encode(text));
        }
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
    },
  });
}

Reading the Stream on the Client

On the browser side you use the Fetch API together with a ReadableStreamDefaultReader to consume the stream incrementally. The key steps are:

  1. Call fetch() — do not await response.json() (that buffers everything).
  2. Get the reader: response.body!.getReader().
  3. Loop with reader.read() until done === true.
  4. Decode each Uint8Array chunk with TextDecoder and append to state.

This is called a pull-based read loop. You pull the next chunk only after you have processed the previous one, giving you natural backpressure.

// Standalone client utility — no React dependency
async function streamCompletion(
  prompt: string,
  onChunk: (text: string) => void
): Promise<void> {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`HTTP ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    const text = decoder.decode(value, { stream: true });
    onChunk(text);
  }
}

// Usage example (TypeScript, no browser DOM required for the logic):
// streamCompletion('Hello!', (chunk) => process.stdout.write(chunk));

Abort Handling with AbortController

Users frequently stop a generation mid-stream. Without abort handling, the server keeps calling the LLM and the client leaks a reader that never closes.

The fix is AbortController. You pass its signal to fetch(). When you call controller.abort(), the fetch throws an AbortError and the browser cancels the underlying TCP read, which in turn triggers backpressure cancellation upstream.

  • Always wrap the read loop in a try/finally block so reader.releaseLock() is always called.
  • On the server, pass the request's signal to the OpenAI SDK so the LLM call itself is cancelled — this saves tokens and money.
// Client-side abort-aware stream reader
async function streamWithAbort(
  prompt: string,
  onChunk: (text: string) => void,
  signal: AbortSignal
): Promise<void> {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
    signal, // <-- pass AbortSignal to fetch
  });

  if (!response.ok || !response.body) {
    throw new Error(`HTTP ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      onChunk(decoder.decode(value, { stream: true }));
    }
  } finally {
    reader.releaseLock(); // always release even on abort
  }
}

Propagating Abort to the LLM on the Server

When the client aborts, the fetch connection closes. Next.js 15 Route Handlers expose req.signal — a native AbortSignal that fires when the client disconnects. Pass this signal to the LLM SDK to cancel the upstream API call immediately.

Inside the ReadableStream constructor you can also implement a cancel() method that cleans up any ongoing work when the stream is cancelled by the consumer.

// app/api/chat/route.ts — with server-side abort propagation
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request): Promise<Response> {
  const { prompt } = await req.json() as { prompt: string };

  const completion = await openai.chat.completions.create(
    {
      model: 'gpt-4o-mini',
      stream: true,
      messages: [{ role: 'user', content: prompt }],
    },
    { signal: req.signal } // propagate client disconnect signal
  );

  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of completion) {
          if (req.signal.aborted) break;
          const text = chunk.choices[0]?.delta?.content ?? '';
          if (text) controller.enqueue(encoder.encode(text));
        }
      } catch (err) {
        if ((err as Error).name !== 'AbortError') throw err;
      } finally {
        controller.close();
      }
    },
    cancel() {
      // Called when the consumer (browser) cancels the stream
      completion.controller.abort();
    },
  });

  return new Response(stream, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Using the Vercel AI SDK for Simpler Streaming

The Vercel AI SDK (ai package) removes most of the boilerplate. streamText() handles the stream construction, abort propagation, and error boundaries for you. It also adds a structured data stream format that the client-side useChat hook can parse.

The result object exposes result.toDataStreamResponse() which returns a fully-formed Response ready to return from your Route Handler.

// app/api/chat/route.ts — Vercel AI SDK
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

export async function POST(req: Request): Promise<Response> {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o-mini'),
    messages,
    abortSignal: req.signal, // automatic abort propagation
  });

  // toDataStreamResponse() streams tokens in Vercel AI data-stream format
  return result.toDataStreamResponse();
}

The useChat Hook on the Client

When the server uses the Vercel AI SDK data-stream format, the companion useChat hook on the client handles everything: fetching, reading the stream, appending tokens, and exposing an isLoading flag plus a stop() function for abort.

This is the recommended pattern for chat UIs in Next.js 15 App Router projects. The hook is framework-agnostic enough to work in both Client Components and Server Components that hydrate client islands.

// app/chat/page.tsx — Client Component using useChat
'use client';

import { useChat } from 'ai/react';

export default function ChatPage() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, stop } =
    useChat({ api: '/api/chat' });

  return (
    <main style={{ maxWidth: 600, margin: '0 auto', padding: 24 }}>
      <ul>
        {messages.map((m) => (
          <li key={m.id}>
            <strong>{m.role}:</strong> {m.content}
          </li>
        ))}
      </ul>
      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} placeholder="Ask anything..." />
        <button type="submit" disabled={isLoading}>Send</button>
        {isLoading && (
          <button type="button" onClick={stop}>Stop</button>
        )}
      </form>
    </main>
  );
}

Backpressure: Why It Matters

Backpressure is the mechanism that prevents a fast producer (the LLM API) from overwhelming a slow consumer (the browser rendering pipeline or a slow network).

In the Web Streams API, backpressure is built in:

  • Each call to reader.read() waits for the consumer to be ready before requesting the next chunk from the underlying source.
  • The ReadableStream internal queue holds a limited number of chunks (controlled by QueuingStrategy). When the queue is full, the producer is paused automatically.
  • If you use for await...of on a stream, the JavaScript runtime handles pull-pacing for you — you get one chunk per iteration, naturally throttled.

You rarely need to configure a custom QueuingStrategy for AI token streams because LLM responses are slow enough that the default high-watermark (1 chunk) is sufficient.

// Demonstrating a custom ByteLengthQueuingStrategy (illustrative)
const strategy = new ByteLengthQueuingStrategy({ highWaterMark: 1024 }); // 1 KB buffer

const stream = new ReadableStream(
  {
    start(controller) {
      // producer: enqueue only when consumer is ready
      controller.enqueue(new TextEncoder().encode('Hello '));
      controller.enqueue(new TextEncoder().encode('world!'));
      controller.close();
    },
  },
  strategy
);

// Consumer: pull-based, respects backpressure
const reader = stream.getReader();
async function drain() {
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    console.log(new TextDecoder().decode(value));
  }
}
drain();

Server-Sent Events vs Raw Streaming

There are two common wire formats for streaming AI responses to the browser:

  • Raw byte stream (Content-Type: text/plain): the simplest approach. Each chunk is raw UTF-8 text. Used in the earlier examples. Works well for simple prose but carries no metadata (token counts, finish reasons, etc.).
  • Server-Sent Events (SSE) (Content-Type: text/event-stream): a standardised line-based format. Each event is data: <payload>\n\n. Browsers have a built-in EventSource API, but it only supports GET. For POST-based chat you use a custom SSE parser on top of fetch.

The Vercel AI SDK data-stream format is an SSE-like protocol that encodes token deltas, tool calls, and metadata as structured lines. Use raw streaming for simple use-cases; use the AI SDK protocol when you need structured events.

// app/api/chat-sse/route.ts — manual SSE format over POST
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request): Promise<Response> {
  const { prompt } = await req.json() as { prompt: string };
  const encoder = new TextEncoder();

  const sseStream = new ReadableStream({
    async start(controller) {
      const completion = await openai.chat.completions.create({
        model: 'gpt-4o-mini',
        stream: true,
        messages: [{ role: 'user', content: prompt }],
      });

      for await (const chunk of completion) {
        const text = chunk.choices[0]?.delta?.content ?? '';
        if (text) {
          // SSE line: data: <payload>\n\n
          controller.enqueue(encoder.encode(`data: ${JSON.stringify({ text })}\n\n`));
        }
      }
      controller.enqueue(encoder.encode('data: [DONE]\n\n'));
      controller.close();
    },
  });

  return new Response(sseStream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
    },
  });
}

Error Handling and Finish Reasons

Streaming introduces new error scenarios that do not exist with a single-shot response:

  • Mid-stream network drop: the read loop throws; catch it and show a "Connection lost" UI.
  • LLM rate-limit (429): the HTTP response itself is non-2xx; check response.ok before reading the body.
  • Content filter stop: the stream ends normally but the last chunk's finish_reason is 'content_filter' — you must read finish reasons out-of-band or from a final SSE event.
  • Token limit reached: finish_reason === 'length' — inform the user the response was truncated.

Always implement a try/catch around the read loop and display graceful fallback UI. Never leave the stream reader locked on error.

// Robust client-side stream reader with error handling
async function safeStream(
  prompt: string,
  onChunk: (t: string) => void,
  onError: (msg: string) => void,
  signal: AbortSignal
): Promise<void> {
  let reader: ReadableStreamDefaultReader<Uint8Array> | null = null;
  try {
    const res = await fetch('/api/chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ prompt }),
      signal,
    });

    if (!res.ok) {
      onError(`Server error: ${res.status} ${res.statusText}`);
      return;
    }

    reader = res.body!.getReader();
    const dec = new TextDecoder();

    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      onChunk(dec.decode(value, { stream: true }));
    }
  } catch (err) {
    if ((err as Error).name === 'AbortError') return; // user-initiated, silent
    onError((err as Error).message);
  } finally {
    reader?.releaseLock();
  }
}

Knowledge Check: Abort Handling

Consider a Next.js 15 Route Handler that streams an OpenAI completion. A user clicks Stop in the UI, which calls controller.abort() on an AbortController whose signal was passed to fetch(). Which additional step is most critical to prevent wasted LLM tokens and unnecessary server compute?

Lesson Recap

In this lesson you learned how to stream LLM completions token-by-token in a Next.js 15 App Router application:

  • Why stream: reduces perceived latency from seconds to milliseconds by sending the first token before the full response is ready.
  • Route Handler: return a new Response(readableStream) from a POST handler; use ReadableStream with controller.enqueue() for each token chunk.
  • Client reader: use response.body.getReader() with a pull-based while (true) { reader.read() } loop and TextDecoder for UTF-8 decoding.
  • Abort handling: pass an AbortSignal to fetch() on the client and to the LLM SDK on the server via req.signal to cancel both sides cleanly.
  • Backpressure: the Web Streams pull model naturally throttles the producer; custom QueuingStrategy is rarely needed for token streams.
  • Vercel AI SDK: streamText() + toDataStreamResponse() on the server paired with useChat on the client eliminates most boilerplate and adds structured metadata support.
  • Error resilience: always use try/finally to release the reader lock, check response.ok before reading, and handle mid-stream drops gracefully.

Frequently asked questions

Is the “Streaming AI Responses Token-by-Token” lesson free?

Yes — the full text of “Streaming AI Responses Token-by-Token” is free to read here on the web, and the Next.js 15 Fullstack (App Router + Server Actions) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Next.js 15 Fullstack (App Router + Server Actions) course, upgrade to CoddyKit PRO.

What will I learn in “Streaming AI Responses Token-by-Token”?

Stream LLM completions to the UI with backpressure-aware readers and abort handling. You practise Next.js 15 Fullstack (App Router + Server Actions) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Next.js 15 Fullstack (App Router + Server Actions)?

No prior experience is required. Next.js 15 Fullstack (App Router + Server Actions) on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Streaming AI Responses Token-by-Token” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Next.js 15 Fullstack (App Router + Server Actions) lesson?

Yes. Every Next.js 15 Fullstack (App Router + Server Actions) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Server-Sent Events from Route Handlers
  2. Integrating WebSocket Services in a Serverless World
  3. Streaming AI Responses Token-by-Token
  4. Presence, Cursors, and Live Collaboration State
← Back to Next.js 15 Fullstack (App Router + Server Actions)