Method Chaining with pipe()
Write readable data transformation pipelines using pipe() to chain custom functions alongside native Pandas methods.
Method Chaining with pipe() is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
The Problem with Intermediate Variables
A data cleaning pipeline without pipe() often accumulates many intermediate variables: df1 = clean(df), df2 = transform(df1), df3 = enrich(df2). These variables clutter the namespace, make debugging harder, and tempt developers to reuse them incorrectly. The result is code that is hard to read top-to-bottom as a sequence of transformations.
import pandas as pd
df = pd.read_csv('orders.csv')
# Without pipe — intermediate variables everywhere
df1 = df.dropna(subset=['revenue'])
df2 = df1[df1['quantity'] > 0]
df3 = df2.assign(revenue_per_unit=df2['revenue'] / df2['quantity'])
print(df3.shape)Introducing pipe()
DataFrame.pipe(func) calls func(df) and returns the result, allowing you to chain custom functions in the same way you chain native Pandas methods like .dropna().query(). The key benefit is that every transformation step is explicit and readable left-to-right (or top-to-bottom when formatted with parentheses), mirroring the logical order of the pipeline.
def drop_nulls(df):
return df.dropna(subset=['revenue'])
def filter_positive_qty(df):
return df[df['quantity'] > 0]
def add_revenue_per_unit(df):
return df.assign(revenue_per_unit=df['revenue'] / df['quantity'])
# With pipe — clean chain
df_clean = (df
.pipe(drop_nulls)
.pipe(filter_positive_qty)
.pipe(add_revenue_per_unit)
)
print(df_clean.shape)Passing Arguments Through pipe()
Pass extra arguments to the piped function using keyword arguments after the function name: df.pipe(func, arg1=val1). The function signature must accept df as its first parameter. Parameterised functions make the pipeline configurable: you can change thresholds, column names, or behaviour without modifying the function body, just by changing the pipe call arguments.
def filter_by_region(df, regions):
return df[df['region'].isin(regions)]
def cap_revenue(df, upper):
df = df.copy()
df['revenue'] = df['revenue'].clip(upper=upper)
return df
df_result = (df
.pipe(filter_by_region, regions=['North', 'East'])
.pipe(cap_revenue, upper=1000)
)
print(df_result.shape)Mixing pipe() with Native Methods
The power of pipe() is that it integrates seamlessly with native Pandas methods in the same chain. You can mix .dropna(), .query(), .rename(), and .pipe(custom_func) in any order. This makes the chain both concise (using built-in methods where possible) and flexible (using custom functions where built-ins fall short).
df_result = (
df
.dropna(subset=['revenue', 'order_date'])
.query('quantity > 0')
.rename(columns={'unit_price': 'price'})
.pipe(add_revenue_per_unit)
.reset_index(drop=True)
)
print(df_result.head())Debugging a pipe() Chain
Debugging a long chain can be tricky because you cannot inspect intermediate states by adding a print statement in the middle. One solution is to write a passthrough debug function that prints shape and column info and then returns the DataFrame unchanged. Insert it at any point in the chain to inspect the state at that step without breaking the chain.
def debug(df, label=''):
print(f'[{label}] shape: {df.shape}')
print(f'[{label}] columns: {df.columns.tolist()}')
return df
df_result = (
df
.pipe(drop_nulls)
.pipe(debug, label='after drop_nulls')
.pipe(filter_positive_qty)
.pipe(debug, label='after filter')
)
print('Done')Building a Full Cleaning Pipeline
Combine all cleaning steps into a single pipeline function using pipe(). Wrapping the chain in a function called clean_pipeline(df) makes the pipeline testable as a unit. Call it with a raw DataFrame and receive a clean one. This pattern aligns with the ETL (Extract, Transform, Load) paradigm used in production data engineering.
def clean_pipeline(df):
return (
df
.dropna(subset=['order_id', 'revenue'])
.drop_duplicates(subset=['order_id'])
.query('quantity > 0 and revenue >= 0')
.pipe(add_revenue_per_unit)
.reset_index(drop=True)
)
df_clean = clean_pipeline(df)
print('Clean rows:', len(df_clean))pipe() vs. apply(): Key Differences
pipe(func) passes the entire DataFrame to func and expects a DataFrame (or transformed object) back. apply(func, axis=1) passes one row at a time. Use pipe() for whole-DataFrame transformations that maintain the same shape (or deliberately change it), and use apply() for row or column level calculations. They are complementary, not competing.
# pipe: receives the whole DataFrame
def scale_revenue(df, factor=1.0):
df = df.copy()
df['revenue'] = df['revenue'] * factor
return df
# apply: receives one row at a time
df['revenue_x2'] = df.apply(lambda row: row['revenue'] * 2, axis=1)
df_scaled = df.pipe(scale_revenue, factor=1.1)
print('pipe scales all rows at once; apply does row-by-row')Reusable Pipeline Components
Write each pipeline step as a pure function — no global state, receives a DataFrame, returns a DataFrame. Pure functions are easy to unit-test: call with a small test DataFrame and assert the output shape and column values. A library of tested, reusable pipeline functions dramatically speeds up analysis of new datasets that share similar cleaning requirements.
def normalise_strings(df, cols):
df = df.copy()
for col in cols:
df[col] = df[col].str.strip().str.lower()
return df
def parse_dates(df, cols):
df = df.copy()
for col in cols:
df[col] = pd.to_datetime(df[col])
return df
df_result = (
df
.pipe(normalise_strings, cols=['region', 'category'])
.pipe(parse_dates, cols=['order_date'])
)
print(df_result.dtypes)Logging Pipeline Steps with pipe()
Add structured logging inside each pipeline function so you can audit every transformation in production. Include the input row count, output row count, and any relevant stats (e.g. rows dropped by a filter). This gives you a complete trace of every pipeline run without needing an external workflow orchestrator for basic audit trails.
import logging
logging.basicConfig(level=logging.INFO)
def logged_dropna(df, subset):
before = len(df)
df = df.dropna(subset=subset)
after = len(df)
logging.info(f'dropna: {before - after} rows removed, {after} remaining')
return df
df_clean = df.pipe(logged_dropna, subset=['revenue'])
print('Logged pipeline complete.')Conditional Steps in a Pipeline
Sometimes a cleaning step should only run based on a flag or the data content. You can add conditional steps to a pipe chain by inserting an identity-or-transform function: it checks a condition and either applies a transformation or returns the DataFrame unchanged. This keeps the chain structure intact while supporting optional steps.
def maybe_cap_revenue(df, cap=None):
if cap is None:
return df
df = df.copy()
df['revenue'] = df['revenue'].clip(upper=cap)
return df
CAPPING_ENABLED = True
CAP_VALUE = 1000 if CAPPING_ENABLED else None
df_result = df.pipe(maybe_cap_revenue, cap=CAP_VALUE)
print('Conditional step applied:', CAPPING_ENABLED)Exporting the Pipeline Result
The final step in a pipe() chain is often an export. You can chain a custom export function using pipe() or simply call the native Pandas export method after the chain. Using a pipe(save_to_parquet) step keeps the export part of the chain documentation and ensures it always runs on the final cleaned DataFrame, not an intermediate version.
def save_parquet(df, path):
df.to_parquet(path, index=False)
print(f'Saved {len(df)} rows to {path}')
return df # return df so the chain can continue if needed
df_final = (
df
.pipe(clean_pipeline)
.pipe(save_parquet, path='orders_final.parquet')
)
print('Pipeline complete.')Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: using pipe() to chain custom functions with native Pandas methods, building reusable, parameterised, and testable pipeline steps as pure functions, and adding debug logging and conditional steps inside a pipe chain. Next up we explore structuring transformation steps as functions for a production ETL pipeline.
Frequently asked questions
Is the “Method Chaining with pipe()” lesson free?
Yes — the full text of “Method Chaining with pipe()” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Method Chaining with pipe()”?
Write readable data transformation pipelines using pipe() to chain custom functions alongside native Pandas methods. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Method Chaining with pipe()” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- apply() on Columns and Rows
- apply() with GroupBy
- map() and applymap() for Element-Wise Operations
- Method Chaining with pipe()