Testing Pipeline Steps with Assertions

A data pipeline that runs without errors can still produce silently wrong output: rows dropped by the wrong filter, a column multiplied by the wrong factor, or a merge that duplicates rows because the join key was not unique. The only way to catch these silent failures is to embed assertions that verify expected properties of the data at each pipeline stage. Assertions turn logical bugs into loud, immediately visible errors.

After every filtering step, assert that the resulting row count is within an expected range. Too few rows means the filter was too aggressive; too many rows means a join duplicated records. Express the expected range as a fraction of the input: for example, a null-drop step should never remove more than 30 % of rows in healthy data. This guard catches unexpected data quality changes in upstream sources.

Assert that all expected output columns exist after every transformation function. If a rename step accidentally uses the wrong key, the resulting column is missing — and the error only surfaces much later when another step tries to use it. An assertion immediately after the transform catches the problem at the source rather than three steps later with a confusing KeyError.

After computing a revenue column, assert it is non-negative. After parsing dates, assert they fall within the expected year range. After encoding categories, assert no unexpected codes appear. Each assertion is a data contract that documents expected behaviour and catches violations early. Write them as assertions rather than print statements so they raise errors in automated pipeline runs.

A common data bug is a many-to-many merge that accidentally multiplies rows. After every pd.merge(), assert that the row count is expected — typically that it did not exceed the left DataFrame's row count for a left join. Also assert that the key column is unique if it should be, so you catch accidental cross-joins immediately.

Certain columns must never be null at any point in the pipeline. Assert df['key_col'].notna().all() after every step that could accidentally introduce nulls — such as a merge that fails to match some rows (producing NaN in joined columns), or a map() call that returns NaN for unmapped values. Make these assertions the last two lines of every transformation function that touches those columns.

Write unit tests for each pipeline function using tiny, hand-crafted DataFrames that isolate the logic being tested. Each test should: set up a minimal input, call the function, and assert the output properties. Using Python's built-in assert is sufficient for simple pipelines; for larger projects, use pytest to run all tests automatically before deploying.

Good tests cover not just the happy path but also edge cases: all-null input, empty DataFrame, single-row DataFrame, and columns with extreme values. An empty DataFrame should return an empty DataFrame, not an error. A DataFrame with a single row should compute the correct result. Test these cases explicitly to ensure the pipeline handles them gracefully in production when unusual data arrives.

Beyond unit tests on individual functions, write an integration test that runs the complete pipeline on a small representative sample of real data. Assert that the output has the expected number of columns, the key column is unique, and the total revenue is within a plausible range. This end-to-end check catches bugs in the interaction between steps that unit tests do not reveal.

As the pipeline grows, collect all tests into a tests/ directory and run them with pytest from the command line. A conftest.py file can create shared fixtures like sample DataFrames. Integrate pytest into your CI/CD pipeline so every code change automatically runs all tests before deployment. A failing test blocks the deployment, preventing broken code from reaching production.

Each assertion in the pipeline is both a guard and a piece of documentation: it states, in machine-readable form, what the data must look like at that point. When a new analyst joins the project and reads the pipeline code, the assertions tell them the data contracts without requiring a separate specification document. Treat every assertion as you would a comment — make the message descriptive enough to understand the business rule it enforces.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: embedding row count, column existence, value range, and null checks as in-pipeline assertions, writing unit tests for individual transformation functions with edge-case coverage, and running integration tests and treating assertions as living data contract documentation. Next up we explore scheduling and logging pipeline runs for automated daily execution.