Validación de esquemas y aserciones

A data pipeline processes new data automatically, often without human review. If the schema of the input file changes — a column is renamed, a date format shifts, or a new category appears — the pipeline should fail loudly rather than produce silently wrong output. Schema validation with assertions is the mechanism that enforces contracts between data producers and data consumers, catching problems at the earliest possible moment.

The most fundamental validation is checking that all required columns are present. Store the expected column set in a config and assert that it is a subset of the actual columns. This check catches renames and drops immediately at pipeline start, before any downstream code attempts to access missing columns and raises a confusing KeyError deep in the pipeline.

After required columns, validate their data types. A date column loaded as object means pd.to_datetime() was not called. An ID column stored as float instead of int64 often means there are NaN values that prevented integer storage. Write type assertions using assert df['col'].dtype == expected_type for the most critical columns.

Key columns like order_id, order_date, and revenue should never be null. Assert df['col'].notna().all() for each of them. To make the assertion message actionable, include the count of null values so the pipeline operator knows the scale of the problem rather than just that an assertion failed.

Numeric columns often have valid business ranges. Revenue must be non-negative. Quantity must be a positive integer. Unit price must be greater than zero. Assert these constraints explicitly — a negative revenue row that slips through will undercount totals in every downstream aggregation without any error. Range checks catch data entry errors and upstream system bugs early.

After deduplication, order_id should be unique. Assert df['order_id'].is_unique to verify this invariant is maintained across every pipeline run. Uniqueness violations after deduplication indicate a bug in the deduplication logic or a newly introduced data source that was not cleaned before merging into the main dataset.

For columns with a finite set of valid values — like region or category — assert that every value is in the allowed set. This catches rogue values that appear after a system migration or an upstream data entry change. Define the valid sets in your pipeline config so they are easy to update when the business expands into new regions.

Validate that all dates fall within the expected period for the dataset. An order dated in the future is impossible; an order dated before the company was founded indicates a corrupted record. Define MIN_DATE and MAX_DATE in the config and assert the column falls within bounds. This also catches Unix-epoch-zero dates (1970-01-01) from bad timestamp conversions.

A sudden change in row count from the previous pipeline run is a strong signal of an upstream issue. Store the previous run's row count in a config file and assert that the new count is within a tolerance band — for example, within ±20 % of the historical count. A dataset that loses 50 % of its rows between runs almost certainly indicates a broken data export.

Collect all assertions into a single validate(df) function that can be called at the start of every pipeline stage. Each check raises an AssertionError with a descriptive message if it fails. This pattern makes the pipeline self-documenting: the validation function is the machine-readable schema contract for the DataFrame at that stage.

In production pipelines, a hard assert crash is acceptable during development but undesirable in a scheduled job. Replace bare assertions with try/except AssertionError blocks that log the error message and optionally send an alert before exiting. This lets the monitoring system capture the failure reason rather than just an unhandled exception traceback.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: asserting required columns, dtypes, non-null constraints, and range validity, checking unique keys, categorical values, and date ranges, and packaging all checks into a reusable validate() function with logging. Next up we explore custom aggregations using apply() on columns and rows.