Go Academy · Lesson

Streaming JSON and CSV

json.Decoder, encoding/csv for large data

Streaming JSON and CSV is a free Go Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Go Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why streaming?

Loading large JSON or CSV files entirely into memory causes high heap usage. Streaming processes records one at a time, keeping memory flat regardless of file size.

json.Decoder for streaming

json.NewDecoder wraps any io.Reader. Use Token() to walk the JSON token stream or Decode inside a loop for JSON arrays.

dec := json.NewDecoder(file)
// consume opening [
for dec.More() {
    var record Record
    dec.Decode(&record)
    process(record)
}

Detecting array start

Call dec.Token() to consume the opening bracket before the decode loop:

t, _ := dec.Token() // consume [
for dec.More() {
    var r Record
    dec.Decode(&r)
}

Streaming from HTTP

Decode JSON directly from an HTTP response body without buffering the entire response:

resp, _ := http.Get(url)
defer resp.Body.Close()
dec := json.NewDecoder(resp.Body)
for dec.More() {
    var item Item
    dec.Decode(&item)
    process(item)
}

encoding/csv reader

csv.NewReader reads CSV row by row. Set LazyQuotes, TrimLeadingSpace, and Comment to handle real-world CSV variants.

r := csv.NewReader(file)
for {
    record, err := r.Read()
    if err == io.EOF { break }
    if err != nil { return err }
    process(record)
}

Skipping the CSV header

Read and discard the first record to skip the header row:

r.Read() // discard header
for {
    record, err := r.Read()
    if err == io.EOF { break }
}

csv.Writer

Write CSV row by row with csv.NewWriter. Always call w.Flush() and check w.Error() after writing.

w := csv.NewWriter(file)
w.Write([]string{"name", "age"})
w.Flush()
if err := w.Error(); err != nil { return err }

Custom delimiter

CSV files sometimes use semicolons or tabs. Set r.Comma to the desired rune.

r := csv.NewReader(file)
r.Comma = ';' // semicolon-separated

Streaming JSON lines (NDJSON)

Newline-delimited JSON (one JSON object per line) is simple to stream: read line by line and unmarshal each line independently.

scanner := bufio.NewScanner(file)
for scanner.Scan() {
    var record Record
    json.Unmarshal(scanner.Bytes(), &record)
}

Memory profile streaming

With streaming, heap allocation stays at O(record size) rather than O(file size). For a 1 GB file of 1 KB records, streaming uses ~1 KB versus ~1 GB for in-memory loading.

Context cancellation in streaming

Check ctx.Done() inside the decode loop to abort streaming when the context is cancelled, preventing goroutines from reading until EOF.

for dec.More() {
    select {
    case <-ctx.Done():
        return ctx.Err()
    default:
    }
    dec.Decode(&record)
}

Quick Check

What is the advantage of streaming JSON with json.Decoder over json.Unmarshal?

Recap: Streaming JSON and CSV

Key points:

json.Decoder: loop with dec.More() for JSON arrays
csv.NewReader: row-by-row; set Comma for custom delimiters
NDJSON: bufio.Scanner + per-line Unmarshal
Streaming keeps memory flat regardless of file size

Frequently asked questions

Is the “Streaming JSON and CSV” lesson free?

Yes — the full text of “Streaming JSON and CSV” is free to read here on the web, and the Go Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Go Academy course, upgrade to CoddyKit PRO.

What will I learn in “Streaming JSON and CSV”?

json.Decoder, encoding/csv for large data You practise Go Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Go Academy?

No prior experience is required. Go Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Streaming JSON and CSV” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Go Academy lesson?

Yes. Every Go Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Go Academy