Streaming JSON and CSV
json.Decoder, encoding/csv for large data
Why streaming?
Loading large JSON or CSV files entirely into memory causes high heap usage. Streaming processes records one at a time, keeping memory flat regardless of file size.
json.Decoder for streaming
json.NewDecoder wraps any io.Reader. Use Token() to walk the JSON token stream or Decode inside a loop for JSON arrays.
dec := json.NewDecoder(file)
// consume opening [
for dec.More() {
var record Record
dec.Decode(&record)
process(record)
}All lessons in this course
- Reading Files with os and bufio
- Writing Files and Temp Files
- JSON Encoding and Decoding
- Streaming JSON and CSV