Incremental Aggregation Across Chunks
Accumulate running counts, sums, and min/max across chunks without storing the full file in memory.
Incremental Aggregation Across Chunks is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Incremental Aggregation?
Incremental aggregation is the key to analysing datasets larger than RAM without distributing computation across multiple machines. Instead of loading all data to compute a final statistic, you maintain running accumulators — partial sums, counts, min/max values — updating them with each chunk. The final result is assembled from these lightweight accumulators after the full file is scanned. This pattern scales to terabyte files on a single laptop.
Counting Rows and Computing Mean
Computing the mean across chunks requires tracking the running sum and count separately. You cannot simply average the per-chunk means because chunks may have different sizes. The correct formula is total_sum / total_count. This pattern extends to any quantity that can be decomposed: variance, correlation, and histograms all have incremental formulas.
import pandas as pd
total_sum = 0.0
total_count = 0
for chunk in pd.read_csv('transactions.csv', chunksize=100000):
total_sum += chunk['amount'].sum()
total_count += chunk['amount'].notna().sum()
grand_mean = total_sum / total_count
print(f'Rows processed: {total_count:,}')
print(f'Grand mean: {grand_mean:.4f}')Incremental Min and Max
Tracking the global minimum and maximum across chunks is straightforward: initialise with Python's float('inf') and float('-inf'), then update with each chunk's min/max. This avoids any intermediate storage. The pattern generalises to per-group min/max by maintaining a dictionary keyed by group identifier.
import pandas as pd
global_min = float('inf')
global_max = float('-inf')
for chunk in pd.read_csv('prices.csv', chunksize=50000):
chunk_min = chunk['price'].min()
chunk_max = chunk['price'].max()
if chunk_min < global_min:
global_min = chunk_min
if chunk_max > global_max:
global_max = chunk_max
print(f'Price range: {global_min} to {global_max}')Incremental Frequency Counts
For categorical columns, maintain a running frequency dictionary by adding each chunk's value_counts() result to a Pandas Series accumulator. Because Pandas Series addition aligns on index labels, unknown categories in later chunks are automatically included. After all chunks, sort by count to see the top categories across the entire dataset.
import pandas as pd
freq = pd.Series(dtype='int64')
for chunk in pd.read_csv('orders.csv',
chunksize=100000,
usecols=['category']):
chunk_counts = chunk['category'].value_counts()
freq = freq.add(chunk_counts, fill_value=0)
# Final sorted frequency table
print(freq.sort_values(ascending=False).head(10))Incremental GroupBy Aggregation
To compute a groupby sum or count across chunks, apply groupby().agg() within each chunk and store the resulting Series or DataFrame. After the loop, concatenate all partial results and apply a second groupby to combine them. This two-stage approach correctly handles groups that appear in multiple chunks, which is common when data is sorted by date rather than by group.
import pandas as pd
partials = []
for chunk in pd.read_csv('sales.csv',
chunksize=100000,
usecols=['region', 'product', 'revenue']):
p = chunk.groupby(['region', 'product'])['revenue'].sum()
partials.append(p)
final = (
pd.concat(partials)
.groupby(level=['region', 'product'])
.sum()
.sort_values(ascending=False)
)
print(final.head(10))Computing Variance Incrementally (Welford's Method)
Computing variance across chunks is trickier than mean. The naive formula E[X²] - E[X]² suffers from catastrophic cancellation for large means. Welford's online algorithm maintains a running mean and sum-of-squared-deviations, updating them with each new value in a numerically stable way. While SciPy implements this, understanding the pattern lets you extend it to weighted variance and covariance.
import pandas as pd
import numpy as np
# Simple two-pass approach using stored chunk stats
chunk_stats = []
for chunk in pd.read_csv('data.csv',
chunksize=100000,
usecols=['value']):
n = chunk['value'].count()
mean = chunk['value'].mean()
var = chunk['value'].var(ddof=1)
chunk_stats.append((n, mean, var))
# Combine: use pooled variance formula
total_n = sum(s[0] for s in chunk_stats)
total_mean = sum(s[0]*s[1] for s in chunk_stats) / total_n
pooled_var = sum((s[0]-1)*s[2] + s[0]*(s[1]-total_mean)**2
for s in chunk_stats) / (total_n - 1)
print(f'Grand variance: {pooled_var:.4f}')Building an Incremental Histogram
Computing the distribution of a column across a file too large for RAM requires an incremental histogram. Fix the bin edges upfront (based on a small sample or domain knowledge), then use np.histogram(chunk_values, bins=edges) within each chunk and accumulate the counts. At the end, plot the combined counts as a bar chart. This is how streaming systems like Kafka Streams and Flink compute approximate histograms.
import pandas as pd
import numpy as np
# Decide bin edges from a sample
sample = pd.read_csv('amounts.csv', nrows=5000)
bins = np.linspace(sample['amount'].min(),
sample['amount'].max(), 21)
counts = np.zeros(len(bins) - 1, dtype='int64')
for chunk in pd.read_csv('amounts.csv',
chunksize=100000,
usecols=['amount']):
chunk_counts, _ = np.histogram(
chunk['amount'].dropna(), bins=bins
)
counts += chunk_counts
print('Histogram counts:', counts[:5], '...')Tracking Unique Values Approximately
Counting exact distinct values across chunks requires storing all unique values — potentially millions. For approximate counts at scale, use a HyperLogLog sketch, available in Python via the hyperloglog library. Alternatively, track uniques per chunk with a set and take the union, but this grows unboundedly. For a cheap approximation, use pd.Series.nunique() per chunk and report the average — not exact, but often sufficient for data profiling.
import pandas as pd
unique_ids = set()
for chunk in pd.read_csv('events.csv',
chunksize=100000,
usecols=['user_id']):
unique_ids.update(chunk['user_id'].dropna().unique())
print(f'Distinct user IDs: {len(unique_ids):,}')
# Warning: the set may grow large for high-cardinality columnsProgress Reporting During Long Runs
Processing a multi-gigabyte file can take minutes. Add progress reporting so you know the pipeline is running and can estimate remaining time. Count bytes or rows processed and compare against the file size. The tqdm library makes this trivial with its tqdm(reader) wrapper. Even without tqdm, printing a status line every 10 chunks gives valuable feedback during long batch jobs.
import pandas as pd
import time
chunksize = 100000
start = time.time()
rows_processed = 0
for i, chunk in enumerate(pd.read_csv('big.csv',
chunksize=chunksize)):
rows_processed += len(chunk)
# Report every 10 chunks
if (i + 1) % 10 == 0:
elapsed = time.time() - start
rate = rows_processed / elapsed
print(f'Chunk {i+1}: {rows_processed:,} rows '
f'@ {rate/1000:.0f}k rows/sec')
print(f'Total: {rows_processed:,} rows in {time.time()-start:.1f}s')Filtering Before Aggregating
Apply filters within each chunk before aggregating to avoid accumulating unwanted data. For example, if you only care about 2024 orders, filter the chunk's date column before groupby. This reduces the memory required for partial results and speeds up the final concatenation step. Always push filters as early as possible in the pipeline — a fundamental principle of efficient data processing.
import pandas as pd
partials = []
for chunk in pd.read_csv('orders.csv',
chunksize=100000,
parse_dates=['order_date']):
# Filter early: only 2024 orders
mask = chunk['order_date'].dt.year == 2024
filtered = chunk.loc[mask, ['category', 'revenue']]
if len(filtered) > 0:
p = filtered.groupby('category')['revenue'].sum()
partials.append(p)
if partials:
result = pd.concat(partials).groupby(level=0).sum()
print(result)Saving Intermediate Results
For very long-running jobs, save intermediate results periodically so you can resume from a checkpoint if the process is interrupted. Write per-chunk aggregates to a Parquet or CSV file after every N chunks. If the job fails at chunk 800 of 1000, you can reload the saved aggregates and continue from where you left off rather than reprocessing the entire file. This resilience pattern is essential in production data pipelines.
import pandas as pd
import os
CHECKPOINT = 'checkpoint.csv'
running_total = 0.0
running_count = 0
# Resume from checkpoint if it exists
if os.path.exists(CHECKPOINT):
ckpt = pd.read_csv(CHECKPOINT)
running_total = ckpt['total'].iloc[0]
running_count = int(ckpt['count'].iloc[0])
print(f'Resuming from checkpoint: {running_count:,} rows')
for chunk in pd.read_csv('huge.csv', chunksize=100000):
running_total += chunk['value'].sum()
running_count += len(chunk)
# Save checkpoint
pd.DataFrame({'total': [running_total],
'count': [running_count]}).to_csv(CHECKPOINT, index=False)
print(f'Final mean: {running_total / running_count:.4f}')Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: running accumulators (sum, count, min/max, frequency Series) enable constant-memory aggregation across large files, two-stage groupby (partial groupby per chunk, then concat and regroup) handles cross-chunk groups correctly, and early filtering within each chunk reduces the cost of the accumulation step. Next up we explore Dask DataFrames as a drop-in parallel replacement for Pandas on large datasets.
Frequently asked questions
Is the “Incremental Aggregation Across Chunks” lesson free?
Yes — the full text of “Incremental Aggregation Across Chunks” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Incremental Aggregation Across Chunks”?
Accumulate running counts, sums, and min/max across chunks without storing the full file in memory. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Incremental Aggregation Across Chunks” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Streaming CSV with chunksize
- Incremental Aggregation Across Chunks
- Introduction to Dask DataFrames
- Parquet: Fast Columnar Storage