Parquet: Schneller spaltenbasierter Speicher

Apache Parquet is a columnar binary file format designed for analytical workloads. Unlike CSV, which stores data row by row as text, Parquet stores each column in a contiguous block, compresses it efficiently, and encodes metadata. This makes it dramatically faster for queries that read only a few columns from a wide table. Parquet is the de-facto standard format in data lakes (AWS S3, Google Cloud Storage) and is natively supported by Pandas, Dask, Spark, and BigQuery.

Converting a Pandas DataFrame to Parquet is a single method call: df.to_parquet('output.parquet'). The default engine is pyarrow (install with pip install pyarrow). Parquet preserves column dtypes exactly — no more date columns being read back as strings. It also supports compression out of the box with the compression parameter; 'snappy' offers fast read/write with moderate compression, while 'gzip' gives higher compression at a cost of speed.

pd.read_parquet('output.parquet') reads the file back into a DataFrame. Compared to reading the equivalent CSV, Parquet reading is typically 5-10x faster for wide tables with many columns. Dtypes are preserved exactly — dates remain datetime64, categories remain category, and integers remain int32 or int64 as stored. No dtype inference is needed because the metadata is embedded in the file.

The biggest advantage of columnar storage is column pruning: you can read only specific columns without scanning the rest of the file. Pass a list of column names to the columns parameter. If a 100-column table has 100 MB per column and you only need 3 columns, Parquet reads 3 MB instead of 10 GB. CSV must scan every character to extract any column. This makes Parquet ideal for wide tables in analytical pipelines.

Parquet stores statistics (min/max) for each row group (block of rows) in the file footer. When you apply a filter via filters parameter, the reader skips entire row groups where the filter cannot match — this is called predicate pushdown. For example, filtering dates in a time-sorted Parquet file skips entire month-blocks that are out of range, reading only the relevant portions. The pyarrow engine supports this natively.

Here is a practical comparison of Parquet vs CSV for a 10 million row, 20-column dataset:

• File size: CSV ~2 GB, Parquet (snappy) ~400 MB
• Read time (all columns): CSV ~15s, Parquet ~2s
• Read time (3 columns): CSV ~15s (must scan all), Parquet ~0.3s
• Dtype preservation: CSV loses dtypes, Parquet preserves them
• Human readable: CSV yes, Parquet no (binary)

For production pipelines where data is written once and read many times, Parquet is almost always the better choice.

For very large datasets, Parquet supports partitioned datasets: the data is split into multiple files organised in a directory hierarchy by column values. For example, partitioning by year and month creates data/year=2024/month=01/part.parquet. Reading a partitioned dataset automatically filters directories matching a query. This is the standard layout in data lakes and enables efficient range queries on billions of rows.

Reading a partitioned Parquet directory is identical to reading a single file — Pandas (via pyarrow) automatically discovers all partition files. The partition column values are included as columns in the resulting DataFrame. You can also use filters to leverage partition pruning, where entire directory subtrees are skipped based on the partition column values. This makes querying a 1 TB partitioned dataset feel like reading a few MB.

Dask natively reads and writes Parquet with dd.read_parquet() and ddf.to_parquet(). Each file in a partitioned Parquet directory becomes one Dask partition, enabling fully parallel reads. Dask also leverages predicate pushdown when filters are specified. Writing a large Dask result to a partitioned Parquet dataset is the standard way to produce outputs in modern data engineering pipelines.

Parquet supports multiple compression algorithms and internal encoding schemes. Snappy (default) prioritises speed with moderate compression. Gzip achieves ~30% smaller files but is slower to read/write. Zstd balances speed and compression better than both. For integer columns, Parquet automatically applies delta encoding or dictionary encoding for further size reduction without any extra configuration on your part.

The simplest way to adopt Parquet is to add a one-time conversion step at the start of a project: read your CSV once, clean and cast dtypes, and save as Parquet. All subsequent runs read the Parquet file instead of the CSV. This gives you immediate speed and storage benefits with minimal code changes. For new data arriving as CSV (e.g., nightly exports), add a conversion step to your pipeline that writes Parquet before any analysis begins.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: to_parquet() and read_parquet() provide faster, smaller, and dtype-preserving file I/O compared to CSV, column pruning with the columns parameter reads only needed columns avoiding full file scans, and partitioned Parquet datasets organised by column values enable partition pruning for efficient range queries on large data lakes. Next up we connect Pandas to relational databases using SQLAlchemy.