PricingLoginStart free
Next.js 15 Fullstack (App Router + Server Actions) · Lezione

Streaming delle risposte AI token per token

Trasmetta le completions degli LLM all’interfaccia con reader consapevoli del backpressure e gestione dell’abort.

Lezione 3 di 413 passaggi

Streaming delle risposte AI token per token è una lezione Next.js 15 Fullstack (App Router + Server Actions) gratuita su CoddyKit. Questa è la lezione 3 di 4. Puoi leggere la lezione completa qui gratuitamente — poi esercitati direttamente nel browser con un editor di codice integrato e un tutor IA disponibile 24/7. Fa parte del percorso di apprendimento Next.js 15 Fullstack (App Router + Server Actions), e i tuoi progressi si sincronizzano tra il web e l'app CoddyKit. Il corso Next.js 15 Fullstack (App Router + Server Actions) include 4 lezioni in totale.

Parti di questa lezione non sono ancora state tradotte e vengono mostrate in inglese.

Why Stream AI Responses?

When you call a large language model (LLM) like GPT-4 or Claude, the model generates tokens one by one. A typical response may take 5–20 seconds to complete. If you wait for the full response before sending anything to the client, the user stares at a blank screen the entire time.

Streaming solves this: you pipe each token to the browser as it is produced, creating the familiar "typewriter" effect used by ChatGPT, Claude.ai, and Gemini.

  • Perceived latency drops from time-to-full-response to time-to-first-token (often under 300 ms).
  • Users can start reading and even abort early if the answer is already clear.
  • Server memory stays flat — you never buffer the whole response.

In Next.js 15 the primitives you need are ReadableStream, the Web Streams API, and StreamingTextResponse (or a plain Response with a stream body).

How LLM SDKs Expose Streams

Most LLM SDKs return an async iterable or a ReadableStream when you pass stream: true. The Vercel AI SDK unifies these under a single interface.

With the official OpenAI SDK you receive a stream of ChatCompletionChunk objects. Each chunk carries a delta.content string that may be one token, a few characters, or an empty string at the end.

  • OpenAI SDK: openai.chat.completions.create({ stream: true }) returns an AsyncIterable.
  • Vercel AI SDK: streamText() returns a result with result.toDataStreamResponse() ready for Next.js Route Handlers.
  • Anthropic SDK: client.messages.stream() returns an async iterable of MessageStreamEvent.

Regardless of the SDK, the pattern is the same: iterate over chunks, encode each piece, and enqueue it into a ReadableStream that becomes the HTTP response body.

Creating a Streaming Route Handler

A Next.js 15 Route Handler returns a standard Response. You can pass a ReadableStream as the body to stream data to the client. The ReadableStream constructor accepts a start(controller) callback where you push chunks with controller.enqueue() and signal completion with controller.close().

Below is a minimal Route Handler at app/api/chat/route.ts that streams an OpenAI completion token by token.

// app/api/chat/route.ts
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request): Promise<Response> {
  const { prompt } = await req.json() as { prompt: string };

  const completion = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    stream: true,
    messages: [{ role: 'user', content: prompt }],
  });

  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();
      for await (const chunk of completion) {
        const text = chunk.choices[0]?.delta?.content ?? '';
        if (text) {
          controller.enqueue(encoder.encode(text));
        }
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
    },
  });
}

Reading the Stream on the Client

On the browser side you use the Fetch API together with a ReadableStreamDefaultReader to consume the stream incrementally. The key steps are:

  1. Call fetch() — do not await response.json() (that buffers everything).
  2. Get the reader: response.body!.getReader().
  3. Loop with reader.read() until done === true.
  4. Decode each Uint8Array chunk with TextDecoder and append to state.

This is called a pull-based read loop. You pull the next chunk only after you have processed the previous one, giving you natural backpressure.

// Standalone client utility — no React dependency
async function streamCompletion(
  prompt: string,
  onChunk: (text: string) => void
): Promise<void> {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
  });

  if (!response.ok || !response.body) {
    throw new Error(`HTTP ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    const text = decoder.decode(value, { stream: true });
    onChunk(text);
  }
}

// Usage example (TypeScript, no browser DOM required for the logic):
// streamCompletion('Hello!', (chunk) => process.stdout.write(chunk));

Abort Handling with AbortController

Users frequently stop a generation mid-stream. Without abort handling, the server keeps calling the LLM and the client leaks a reader that never closes.

The fix is AbortController. You pass its signal to fetch(). When you call controller.abort(), the fetch throws an AbortError and the browser cancels the underlying TCP read, which in turn triggers backpressure cancellation upstream.

  • Always wrap the read loop in a try/finally block so reader.releaseLock() is always called.
  • On the server, pass the request's signal to the OpenAI SDK so the LLM call itself is cancelled — this saves tokens and money.
// Client-side abort-aware stream reader
async function streamWithAbort(
  prompt: string,
  onChunk: (text: string) => void,
  signal: AbortSignal
): Promise<void> {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ prompt }),
    signal, // <-- pass AbortSignal to fetch
  });

  if (!response.ok || !response.body) {
    throw new Error(`HTTP ${response.status}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  try {
    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      onChunk(decoder.decode(value, { stream: true }));
    }
  } finally {
    reader.releaseLock(); // always release even on abort
  }
}

Propagating Abort to the LLM on the Server

When the client aborts, the fetch connection closes. Next.js 15 Route Handlers expose req.signal — a native AbortSignal that fires when the client disconnects. Pass this signal to the LLM SDK to cancel the upstream API call immediately.

Inside the ReadableStream constructor you can also implement a cancel() method that cleans up any ongoing work when the stream is cancelled by the consumer.

// app/api/chat/route.ts — with server-side abort propagation
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request): Promise<Response> {
  const { prompt } = await req.json() as { prompt: string };

  const completion = await openai.chat.completions.create(
    {
      model: 'gpt-4o-mini',
      stream: true,
      messages: [{ role: 'user', content: prompt }],
    },
    { signal: req.signal } // propagate client disconnect signal
  );

  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      try {
        for await (const chunk of completion) {
          if (req.signal.aborted) break;
          const text = chunk.choices[0]?.delta?.content ?? '';
          if (text) controller.enqueue(encoder.encode(text));
        }
      } catch (err) {
        if ((err as Error).name !== 'AbortError') throw err;
      } finally {
        controller.close();
      }
    },
    cancel() {
      // Called when the consumer (browser) cancels the stream
      completion.controller.abort();
    },
  });

  return new Response(stream, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Using the Vercel AI SDK for Simpler Streaming

The Vercel AI SDK (ai package) removes most of the boilerplate. streamText() handles the stream construction, abort propagation, and error boundaries for you. It also adds a structured data stream format that the client-side useChat hook can parse.

The result object exposes result.toDataStreamResponse() which returns a fully-formed Response ready to return from your Route Handler.

// app/api/chat/route.ts — Vercel AI SDK
import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';

export async function POST(req: Request): Promise<Response> {
  const { messages } = await req.json();

  const result = streamText({
    model: openai('gpt-4o-mini'),
    messages,
    abortSignal: req.signal, // automatic abort propagation
  });

  // toDataStreamResponse() streams tokens in Vercel AI data-stream format
  return result.toDataStreamResponse();
}

The useChat Hook on the Client

When the server uses the Vercel AI SDK data-stream format, the companion useChat hook on the client handles everything: fetching, reading the stream, appending tokens, and exposing an isLoading flag plus a stop() function for abort.

This is the recommended pattern for chat UIs in Next.js 15 App Router projects. The hook is framework-agnostic enough to work in both Client Components and Server Components that hydrate client islands.

// app/chat/page.tsx — Client Component using useChat
'use client';

import { useChat } from 'ai/react';

export default function ChatPage() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, stop } =
    useChat({ api: '/api/chat' });

  return (
    <main style={{ maxWidth: 600, margin: '0 auto', padding: 24 }}>
      <ul>
        {messages.map((m) => (
          <li key={m.id}>
            <strong>{m.role}:</strong> {m.content}
          </li>
        ))}
      </ul>
      <form onSubmit={handleSubmit}>
        <input value={input} onChange={handleInputChange} placeholder="Ask anything..." />
        <button type="submit" disabled={isLoading}>Send</button>
        {isLoading && (
          <button type="button" onClick={stop}>Stop</button>
        )}
      </form>
    </main>
  );
}

Backpressure: Why It Matters

Backpressure is the mechanism that prevents a fast producer (the LLM API) from overwhelming a slow consumer (the browser rendering pipeline or a slow network).

In the Web Streams API, backpressure is built in:

  • Each call to reader.read() waits for the consumer to be ready before requesting the next chunk from the underlying source.
  • The ReadableStream internal queue holds a limited number of chunks (controlled by QueuingStrategy). When the queue is full, the producer is paused automatically.
  • If you use for await...of on a stream, the JavaScript runtime handles pull-pacing for you — you get one chunk per iteration, naturally throttled.

You rarely need to configure a custom QueuingStrategy for AI token streams because LLM responses are slow enough that the default high-watermark (1 chunk) is sufficient.

// Demonstrating a custom ByteLengthQueuingStrategy (illustrative)
const strategy = new ByteLengthQueuingStrategy({ highWaterMark: 1024 }); // 1 KB buffer

const stream = new ReadableStream(
  {
    start(controller) {
      // producer: enqueue only when consumer is ready
      controller.enqueue(new TextEncoder().encode('Hello '));
      controller.enqueue(new TextEncoder().encode('world!'));
      controller.close();
    },
  },
  strategy
);

// Consumer: pull-based, respects backpressure
const reader = stream.getReader();
async function drain() {
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    console.log(new TextDecoder().decode(value));
  }
}
drain();

Server-Sent Events vs Raw Streaming

There are two common wire formats for streaming AI responses to the browser:

  • Raw byte stream (Content-Type: text/plain): the simplest approach. Each chunk is raw UTF-8 text. Used in the earlier examples. Works well for simple prose but carries no metadata (token counts, finish reasons, etc.).
  • Server-Sent Events (SSE) (Content-Type: text/event-stream): a standardised line-based format. Each event is data: <payload>\n\n. Browsers have a built-in EventSource API, but it only supports GET. For POST-based chat you use a custom SSE parser on top of fetch.

The Vercel AI SDK data-stream format is an SSE-like protocol that encodes token deltas, tool calls, and metadata as structured lines. Use raw streaming for simple use-cases; use the AI SDK protocol when you need structured events.

// app/api/chat-sse/route.ts — manual SSE format over POST
import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function POST(req: Request): Promise<Response> {
  const { prompt } = await req.json() as { prompt: string };
  const encoder = new TextEncoder();

  const sseStream = new ReadableStream({
    async start(controller) {
      const completion = await openai.chat.completions.create({
        model: 'gpt-4o-mini',
        stream: true,
        messages: [{ role: 'user', content: prompt }],
      });

      for await (const chunk of completion) {
        const text = chunk.choices[0]?.delta?.content ?? '';
        if (text) {
          // SSE line: data: <payload>\n\n
          controller.enqueue(encoder.encode(`data: ${JSON.stringify({ text })}\n\n`));
        }
      }
      controller.enqueue(encoder.encode('data: [DONE]\n\n'));
      controller.close();
    },
  });

  return new Response(sseStream, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      Connection: 'keep-alive',
    },
  });
}

Error Handling and Finish Reasons

Streaming introduces new error scenarios that do not exist with a single-shot response:

  • Mid-stream network drop: the read loop throws; catch it and show a "Connection lost" UI.
  • LLM rate-limit (429): the HTTP response itself is non-2xx; check response.ok before reading the body.
  • Content filter stop: the stream ends normally but the last chunk's finish_reason is 'content_filter' — you must read finish reasons out-of-band or from a final SSE event.
  • Token limit reached: finish_reason === 'length' — inform the user the response was truncated.

Always implement a try/catch around the read loop and display graceful fallback UI. Never leave the stream reader locked on error.

// Robust client-side stream reader with error handling
async function safeStream(
  prompt: string,
  onChunk: (t: string) => void,
  onError: (msg: string) => void,
  signal: AbortSignal
): Promise<void> {
  let reader: ReadableStreamDefaultReader<Uint8Array> | null = null;
  try {
    const res = await fetch('/api/chat', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ prompt }),
      signal,
    });

    if (!res.ok) {
      onError(`Server error: ${res.status} ${res.statusText}`);
      return;
    }

    reader = res.body!.getReader();
    const dec = new TextDecoder();

    while (true) {
      const { value, done } = await reader.read();
      if (done) break;
      onChunk(dec.decode(value, { stream: true }));
    }
  } catch (err) {
    if ((err as Error).name === 'AbortError') return; // user-initiated, silent
    onError((err as Error).message);
  } finally {
    reader?.releaseLock();
  }
}

Knowledge Check: Abort Handling

Consider a Next.js 15 Route Handler that streams an OpenAI completion. A user clicks Stop in the UI, which calls controller.abort() on an AbortController whose signal was passed to fetch(). Which additional step is most critical to prevent wasted LLM tokens and unnecessary server compute?

Rispondi nella lezione interattiva

Lesson Recap

In this lesson you learned how to stream LLM completions token-by-token in a Next.js 15 App Router application:

  • Why stream: reduces perceived latency from seconds to milliseconds by sending the first token before the full response is ready.
  • Route Handler: return a new Response(readableStream) from a POST handler; use ReadableStream with controller.enqueue() for each token chunk.
  • Client reader: use response.body.getReader() with a pull-based while (true) { reader.read() } loop and TextDecoder for UTF-8 decoding.
  • Abort handling: pass an AbortSignal to fetch() on the client and to the LLM SDK on the server via req.signal to cancel both sides cleanly.
  • Backpressure: the Web Streams pull model naturally throttles the producer; custom QueuingStrategy is rarely needed for token streams.
  • Vercel AI SDK: streamText() + toDataStreamResponse() on the server paired with useChat on the client eliminates most boilerplate and adds structured metadata support.
  • Error resilience: always use try/finally to release the reader lock, check response.ok before reading, and handle mid-stream drops gracefully.
Gratis per iniziare

Impara TypeScript con un tutor IA — gratis

Scrivi ed esegui vero codice nel tuo browser, ricevi aiuto istantaneo da un tutor IA disponibile 24/7, e riprendi da dove hai lasciato sul web o nell'app.

Corsi
22
Lezioni
88
Inizia a imparare gratisVisualizza il corso completo

Domande Frequenti

La lezione «Streaming delle risposte AI token per token» è gratuita?

Sì — il testo completo di «Streaming delle risposte AI token per token» è gratuito qui sul web. Per esercitarvi in modo interattivo (un editor di codice integrato e un tutor IA 24/7) e sbloccare il resto del corso Next.js 15 Fullstack (App Router + Server Actions), passa a CoddyKit PRO. Il corso Next.js 15 Fullstack (App Router + Server Actions) include 4 lezioni in totale.

Cosa imparerò in «Streaming delle risposte AI token per token»?

Trasmetta le completions degli LLM all’interfaccia con reader consapevoli del backpressure e gestione dell’abort. Eserciti Next.js 15 Fullstack (App Router + Server Actions) con codice pratico che esegui direttamente nel browser, e un tutor IA 24/7 risponde alle tue domande mentre lavori sulla lezione.

Ho bisogno di esperienza per iniziare Next.js 15 Fullstack (App Router + Server Actions)?

Non è richiesta alcuna esperienza precedente. Next.js 15 Fullstack (App Router + Server Actions) su CoddyKit è strutturato per principianti e studenti avanzati, quindi puoi iniziare da qui o dall'inizio e procedere al tuo ritmo. Questa è la lezione 3 di 4.

Quanto tempo richiede la lezione «Streaming delle risposte AI token per token»?

La maggior parte delle lezioni CoddyKit richiede circa 5–10 minuti. Ogni lezione è breve e interattiva, quindi fai progressi costanti e riprendi esattamente da dove hai lasciato su web e app.

Posso scrivere ed eseguire codice in questa lezione Next.js 15 Fullstack (App Router + Server Actions)?

Sì. Ogni lezione Next.js 15 Fullstack (App Router + Server Actions) include un editor di codice integrato, quindi scrivi ed esegui codice reale direttamente nel tuo browser e ricevi feedback istantaneo dall'IA — nessuna configurazione locale necessaria.

Tutte le lezioni di questo corso

  1. Server-Sent Events dai route handler
  2. Integrazione di servizi WebSocket in un ambiente serverless
  3. Streaming delle risposte AI token per token
  4. Presenza, cursori e stato della collaborazione in tempo reale
← Torna a Next.js 15 Fullstack (App Router + Server Actions)