Go Academy · Lesson

Streaming Large Files

Process files without loading all.

Streaming Large Files is a free Go Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Go Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Problem with Loading All

Reading a multi-gigabyte file into memory with os.ReadFile can exhaust RAM. Instead, stream it: process chunks as they arrive.

Streaming Concept

Streaming means you keep only a small window of data in memory at once. Combined with buffering, you process arbitrarily large inputs with constant memory.

Line-by-Line with Scanner

For text, a bufio.Scanner over a reader processes one line at a time regardless of file size. Here we simulate a file with a strings.Reader.

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func main() {
	src := strings.NewReader("alpha\nbeta\ngamma")
	s := bufio.NewScanner(src)
	for s.Scan() {
		fmt.Println("processed:", s.Text())
	}
}

Counting Without Loading

You can compute statistics over a huge stream while holding almost nothing in memory.

package main

import (
	"bufio"
	"fmt"
	"strings"
)

func main() {
	s := bufio.NewScanner(strings.NewReader("a\nbb\nccc"))
	total := 0
	for s.Scan() {
		total += len(s.Text())
	}
	fmt.Println("total chars:", total)
}

Fixed-Size Chunks

For binary data, read into a fixed buffer with Read. n tells you how many bytes were filled.

package main

import (
	"fmt"
	"io"
	"strings"
)

func main() {
	src := strings.NewReader("0123456789")
	buf := make([]byte, 4)
	for {
		n, err := src.Read(buf)
		if n > 0 {
			fmt.Printf("chunk: %s\n", buf[:n])
		}
		if err == io.EOF {
			break
		}
	}
}

io.Copy for Streaming

io.Copy(dst, src) streams everything from a reader to a writer using a small internal buffer — no full load.

package main

import (
	"io"
	"os"
	"strings"
)

func main() {
	src := strings.NewReader("streamed straight through\n")
	io.Copy(os.Stdout, src)
}

Combining Reader and Writer Buffers

Wrap both ends in bufio for efficient streaming transforms: buffered read, process, buffered write, flush.

package main

import (
	"bufio"
	"os"
	"strings"
)

func main() {
	r := bufio.NewScanner(strings.NewReader("one\ntwo"))
	w := bufio.NewWriter(os.Stdout)
	defer w.Flush()
	for r.Scan() {
		w.WriteString(r.Text() + "!\n")
	}
}

Constant Memory Guarantee

Because the buffer size is fixed, memory use stays flat whether the input is 1KB or 1TB. This is the key advantage of streaming.

Avoiding Common Mistakes

Do not call os.ReadFile on huge files
Always handle n bytes before checking the error
Remember to Flush() buffered writers

Transforming a Stream

Here we uppercase each line as it streams through, keeping only one line in memory at a time.

package main

import (
	"bufio"
	"os"
	"strings"
)

func main() {
	s := bufio.NewScanner(strings.NewReader("hi\nthere"))
	w := bufio.NewWriter(os.Stdout)
	defer w.Flush()
	for s.Scan() {
		w.WriteString(strings.ToUpper(s.Text()) + "\n")
	}
}

When to Stream

Stream when input size is large or unknown, or when you can process incrementally. Load fully only for small, bounded data you need all at once.

Quick Check

Test your streaming knowledge.

Recap

You learned to process large data without loading it all.

Scanner for line-by-line text
Fixed-size Read for binary chunks
io.Copy for stream-to-stream
Buffer both ends, flush writers, constant memory

Frequently asked questions

Is the “Streaming Large Files” lesson free?

Yes — the full text of “Streaming Large Files” is free to read here on the web, and the Go Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Go Academy course, upgrade to CoddyKit PRO.

What will I learn in “Streaming Large Files”?

Process files without loading all. You practise Go Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Go Academy?

No prior experience is required. Go Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Streaming Large Files” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Go Academy lesson?

Yes. Every Go Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Go Academy