Chunked Reading for Large Files

A common bottleneck in production data pipelines is a CSV file that is larger than available RAM. If a file is 20 GB and the machine has 16 GB of RAM, pd.read_csv('file.csv') will crash with a MemoryError. The solution is chunked reading: loading the file in fixed-size pieces, processing each piece, and accumulating results. This allows you to analyse arbitrarily large files without loading them entirely into memory.

Pass chunksize=n to pd.read_csv() to return a TextFileReader iterator instead of a DataFrame. Each iteration yields the next n rows as a DataFrame. This way, only n rows are in memory at a time. A good starting value for chunksize is 100,000 rows — small enough to fit in RAM but large enough to keep I/O overhead manageable. Adjust based on your available RAM and column count.

Use a for loop over the chunk iterator to process each chunk. For simple operations like counting rows, summing a column, or filtering, process each chunk independently and accumulate the results in a list or running total. Always call the iterator inside a with statement or ensure you re-create it for each pipeline run — iterators are consumed once and cannot be restarted.

Apply row filters inside the loop to discard unwanted data before it accumulates. This keeps memory usage low by only retaining rows that match your criteria. For example, to extract all 'North' region rows from a 10 GB file, filter each chunk before appending to your results list. The final pd.concat only handles the filtered subset, which is much smaller.

GroupBy aggregations across chunks are trickier than simple sums because groups may span multiple chunks. The pattern is: compute group-level sums and counts per chunk, concatenate these partial results, and perform a final groupby on the accumulated partial results. Never call groupby().mean() per chunk and then average the averages — this gives wrong results when group sizes differ across chunks.

Reduce memory during chunked loading by specifying column dtypes upfront in read_csv. This prevents Pandas from allocating wide types for each chunk and then discarding them. Pass a dict like dtype={'region': 'category', 'sales': 'int32'} to read_csv(). Combined with chunked reading, this can reduce peak memory to less than 10% of a naive full-file load.

When the output of processing each chunk also needs to be written to a file, write each processed chunk as you go rather than accumulating everything in memory. Use mode='a' (append) and header=False (after the first chunk) with to_csv(). This keeps both input and output memory footprints at chunk size, making the pipeline capable of processing arbitrarily large files on a memory-constrained machine.

Dask provides a Pandas-like API that executes operations lazily and in parallel across chunks automatically. Instead of writing a manual chunking loop, you write dask_df = dd.read_csv('file.csv') and then the same Pandas operations — groupby, filter, merge. Call .compute() at the end to trigger execution. Dask is the most practical solution when your pipeline involves complex multi-step operations that are hard to implement manually with chunking.

The ideal chunk size balances two competing factors: too small (e.g. 1,000 rows) causes high per-chunk overhead (file I/O setup, DataFrame creation) and the loop takes longer. Too large (e.g. 10M rows) defeats the purpose by loading too much data into RAM at once. A practical starting point: target chunks of 50-200 MB in memory. For a DataFrame with 10 columns averaging 8 bytes each, 100,000 rows ≈ 8 MB per chunk — a safe default that leaves plenty of headroom.

If you control the file format, prefer Parquet over CSV for large datasets. Parquet is a columnar binary format that: loads only requested columns (column pruning), compresses much better than CSV, preserves dtypes (no re-inference on load), and reads 5-20x faster than an equivalent CSV. Convert a large CSV to Parquet once with pd.read_csv('file.csv', chunksize=500000) + to_parquet, then use pd.read_parquet(columns=['col1','col2']) for all subsequent reads.

A complete pattern for large file processing: 1) Specify dtypes at read time. 2) Filter rows as early as possible inside the loop. 3) Compute partial aggregates per chunk (sum + count, never mean directly). 4) After the loop, combine partials and compute final statistics. 5) Write output incrementally if results are large. This pattern handles files of any size on any machine with adequate chunksize tuning.

Test your understanding of chunked reading from this lesson.

In this lesson you learned: chunksize in read_csv returns an iterator yielding one DataFrame per chunk, allowing files larger than RAM to be processed incrementally, partial aggregations (sum + count) accumulate correctly across chunks while mean cannot be averaged directly, and Parquet format is the best long-term alternative to CSV for large datasets due to compression, speed, and dtype preservation. This completes the Pandas Performance Tips course — you are ready for the advanced B2 courses!