Pandas & NumPy Academy · Lesson

Parquet: Fast Columnar Storage

Write a Pandas DataFrame to Parquet with to_parquet(), read it back faster than CSV, and use column pruning to load only needed fields.

Parquet: Fast Columnar Storage is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

What Is Parquet?

Apache Parquet is a columnar binary file format designed for analytical workloads. Unlike CSV, which stores data row by row as text, Parquet stores each column in a contiguous block, compresses it efficiently, and encodes metadata. This makes it dramatically faster for queries that read only a few columns from a wide table. Parquet is the de-facto standard format in data lakes (AWS S3, Google Cloud Storage) and is natively supported by Pandas, Dask, Spark, and BigQuery.

Writing Parquet with to_parquet()

Converting a Pandas DataFrame to Parquet is a single method call: df.to_parquet('output.parquet'). The default engine is pyarrow (install with pip install pyarrow). Parquet preserves column dtypes exactly — no more date columns being read back as strings. It also supports compression out of the box with the compression parameter; 'snappy' offers fast read/write with moderate compression, while 'gzip' gives higher compression at a cost of speed.

import pandas as pd
import numpy as np

# Create a sample DataFrame
np.random.seed(0)
df = pd.DataFrame({
    'date': pd.date_range('2024-01-01', periods=100000, freq='T'),
    'value': np.random.randn(100000),
    'category': np.random.choice(['A', 'B', 'C'], 100000)
})

# Write to Parquet
df.to_parquet('data.parquet', index=False, compression='snappy')
print('Written to data.parquet')

Reading Parquet with read_parquet()

pd.read_parquet('output.parquet') reads the file back into a DataFrame. Compared to reading the equivalent CSV, Parquet reading is typically 5-10x faster for wide tables with many columns. Dtypes are preserved exactly — dates remain datetime64, categories remain category, and integers remain int32 or int64 as stored. No dtype inference is needed because the metadata is embedded in the file.

import pandas as pd
import time

# Read Parquet
start = time.time()
df = pd.read_parquet('data.parquet')
print(f'Read Parquet: {time.time()-start:.3f}s')
print(df.dtypes)
print(df.shape)

Column Pruning: Reading Only Needed Columns

The biggest advantage of columnar storage is column pruning: you can read only specific columns without scanning the rest of the file. Pass a list of column names to the columns parameter. If a 100-column table has 100 MB per column and you only need 3 columns, Parquet reads 3 MB instead of 10 GB. CSV must scan every character to extract any column. This makes Parquet ideal for wide tables in analytical pipelines.

import pandas as pd

# Only load the 2 columns needed for this analysis
df = pd.read_parquet('large_table.parquet',
                     columns=['date', 'revenue'])
print(df.columns.tolist())
print(df.shape)
print(df.memory_usage(deep=True).sum() / 1e6, 'MB loaded')

Row Group Filtering (Predicate Pushdown)

Parquet stores statistics (min/max) for each row group (block of rows) in the file footer. When you apply a filter via filters parameter, the reader skips entire row groups where the filter cannot match — this is called predicate pushdown. For example, filtering dates in a time-sorted Parquet file skips entire month-blocks that are out of range, reading only the relevant portions. The pyarrow engine supports this natively.

import pandas as pd

# Filter using predicate pushdown — row groups outside range are skipped
df = pd.read_parquet(
    'time_series.parquet',
    columns=['date', 'value'],
    filters=[('date', '>=', '2024-06-01'),
              ('date', '<', '2024-07-01')]
)
print(f'Loaded {len(df):,} rows (June only)')
print(df.head())

Parquet vs CSV: A Comparison

Here is a practical comparison of Parquet vs CSV for a 10 million row, 20-column dataset:

File size: CSV ~2 GB, Parquet (snappy) ~400 MB
Read time (all columns): CSV ~15s, Parquet ~2s
Read time (3 columns): CSV ~15s (must scan all), Parquet ~0.3s
Dtype preservation: CSV loses dtypes, Parquet preserves them
Human readable: CSV yes, Parquet no (binary)

For production pipelines where data is written once and read many times, Parquet is almost always the better choice.

Partitioned Parquet Datasets

For very large datasets, Parquet supports partitioned datasets: the data is split into multiple files organised in a directory hierarchy by column values. For example, partitioning by year and month creates data/year=2024/month=01/part.parquet. Reading a partitioned dataset automatically filters directories matching a query. This is the standard layout in data lakes and enables efficient range queries on billions of rows.

import pandas as pd

# Write a partitioned dataset (requires pyarrow)
df = pd.read_parquet('all_data.parquet')
df['year'] = df['date'].dt.year
df['month'] = df['date'].dt.month

df.to_parquet(
    'partitioned_data/',
    partition_cols=['year', 'month'],
    index=False
)
# Creates: partitioned_data/year=2024/month=1/part-0.parquet etc.

Reading Partitioned Datasets

Reading a partitioned Parquet directory is identical to reading a single file — Pandas (via pyarrow) automatically discovers all partition files. The partition column values are included as columns in the resulting DataFrame. You can also use filters to leverage partition pruning, where entire directory subtrees are skipped based on the partition column values. This makes querying a 1 TB partitioned dataset feel like reading a few MB.

import pandas as pd

# Read the entire partitioned dataset
df_all = pd.read_parquet('partitioned_data/')
print('All years:', df_all['year'].unique())

# Read only 2024 data using partition pruning
df_2024 = pd.read_parquet(
    'partitioned_data/',
    filters=[('year', '==', 2024)]
)
print('2024 rows:', len(df_2024))

Parquet with Dask

Dask natively reads and writes Parquet with dd.read_parquet() and ddf.to_parquet(). Each file in a partitioned Parquet directory becomes one Dask partition, enabling fully parallel reads. Dask also leverages predicate pushdown when filters are specified. Writing a large Dask result to a partitioned Parquet dataset is the standard way to produce outputs in modern data engineering pipelines.

import dask.dataframe as dd

# Read partitioned Parquet with Dask (each file = one partition)
ddf = dd.read_parquet('partitioned_data/',
                      columns=['date', 'revenue', 'region'],
                      filters=[('year', '==', 2024)])

# Compute aggregation in parallel
result = ddf.groupby('region')['revenue'].sum().compute()
print(result.sort_values(ascending=False))

Compression and Encoding Options

Parquet supports multiple compression algorithms and internal encoding schemes. Snappy (default) prioritises speed with moderate compression. Gzip achieves ~30% smaller files but is slower to read/write. Zstd balances speed and compression better than both. For integer columns, Parquet automatically applies delta encoding or dictionary encoding for further size reduction without any extra configuration on your part.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'id': range(500000), 'val': np.random.randn(500000)})

for comp in ['snappy', 'gzip', 'zstd']:
    fname = f'data_{comp}.parquet'
    df.to_parquet(fname, compression=comp, index=False)
    import os
    size_mb = os.path.getsize(fname) / 1e6
    print(f'{comp}: {size_mb:.2f} MB')

Replacing CSV in Your Pipeline

The simplest way to adopt Parquet is to add a one-time conversion step at the start of a project: read your CSV once, clean and cast dtypes, and save as Parquet. All subsequent runs read the Parquet file instead of the CSV. This gives you immediate speed and storage benefits with minimal code changes. For new data arriving as CSV (e.g., nightly exports), add a conversion step to your pipeline that writes Parquet before any analysis begins.

import pandas as pd

# One-time conversion
df = pd.read_csv('raw_data.csv',
                 parse_dates=['date'],
                 dtype={'category': 'category',
                        'amount': 'float32'})
df.to_parquet('clean_data.parquet', index=False)

# All future reads use Parquet
df_fast = pd.read_parquet('clean_data.parquet')
print('Loaded from Parquet:', df_fast.dtypes.to_dict())

Quick Check

Test your understanding of Data Analysis concepts from this lesson.

Lesson Recap

In this lesson you learned: to_parquet() and read_parquet() provide faster, smaller, and dtype-preserving file I/O compared to CSV, column pruning with the columns parameter reads only needed columns avoiding full file scans, and partitioned Parquet datasets organised by column values enable partition pruning for efficient range queries on large data lakes. Next up we connect Pandas to relational databases using SQLAlchemy.

Frequently asked questions

Is the “Parquet: Fast Columnar Storage” lesson free?

Yes — the full text of “Parquet: Fast Columnar Storage” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.

What will I learn in “Parquet: Fast Columnar Storage”?

Write a Pandas DataFrame to Parquet with to_parquet(), read it back faster than CSV, and use column pruning to load only needed fields. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Pandas & NumPy Academy?

No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Parquet: Fast Columnar Storage” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Pandas & NumPy Academy lesson?

Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Pandas & NumPy Academy