pd.concat for Stacking DataFrames

In real projects, data rarely arrives in a single file. You might have monthly sales files, regional survey exports, or database query results split across multiple queries. Concatenation stacks these separate DataFrames into one combined table. Pandas provides pd.concat() for this purpose, handling both vertical (row-wise) and horizontal (column-wise) stacking.

The most common use of pd.concat() is stacking DataFrames vertically (adding more rows). Pass a list of DataFrames and the function appends them one below the other. Both DataFrames must have compatible columns for a clean result. Pandas aligns on column names automatically, filling missing columns with NaN.

Notice that pd.concat() preserves the original indices from each DataFrame, which can result in duplicate index values (as seen above with two rows at index 0 and index 1). Pass ignore_index=True to create a fresh sequential integer index in the combined result, which is almost always what you want.

When concatenating data from multiple sources, you may want to know which original DataFrame each row came from. Pass the keys parameter with a list of labels. Pandas creates a MultiIndex where the outer level identifies the source. You can then use .loc['label'] to access rows from a specific source.

Pass axis=1 to concatenate DataFrames side by side (adding more columns). Pandas aligns on the row index, so both DataFrames should have the same index for a clean result. If indices differ, non-matching rows will be filled with NaN. This is useful for combining features computed from the same dataset in separate steps.

If the DataFrames being concatenated have different columns, Pandas keeps all columns and fills missing values with NaN. This is the default join='outer' behaviour. If you want to keep only the columns that appear in all DataFrames, pass join='inner', which drops any column not present in every source.

A typical pattern when loading multiple files is to collect all DataFrames in a list and call pd.concat() once at the end. Avoid concatenating inside a loop (e.g., df = pd.concat([df, new_chunk])) because this creates a new DataFrame copy on every iteration, leading to quadratic time complexity for large collections.

pd.concat() works with Series as well as DataFrames. Concatenating a list of Series vertically gives a longer Series. Concatenating with axis=1 produces a DataFrame where each Series becomes a column. The Series are aligned on their index, so the index labels must match for a clean horizontal concat.

After concatenating, always verify the result has the expected shape and no unexpected NaN values. A quick sanity check pattern: compare the combined row count to the sum of individual counts, and call isna().sum() to spot any unintended missing values introduced by column misalignment. These checks prevent silent data quality issues from propagating into your analysis.

Older Pandas code may use df.append(other), which was a convenience wrapper around pd.concat(). This method was deprecated in Pandas 1.4 and removed in Pandas 2.0. Always use pd.concat([df, other], ignore_index=True) in modern code. The behaviour is identical but pd.concat is more explicit and supports concatenating more than two DataFrames at once.

Here is a realistic pattern: load monthly DataFrames, add a source label, concatenate, and compute a full-year summary. The keys parameter makes it easy to trace each row back to its month, and ignore_index=True combined with a month column gives a clean, flat DataFrame for group-by analysis.

Test your understanding of pd.concat for stacking DataFrames from this lesson.

In this lesson you learned: how to stack DataFrames vertically with pd.concat() and ignore_index=True, how to track sources using the keys parameter, and how join='inner' vs 'outer' controls column handling when schemas differ. Next up we explore pd.merge() for SQL-style inner and outer joins on shared key columns.