模式验证与断言
针对列范围、非空约束和唯一键编写断言检查，并让它们在每个 pipeline 开始时运行。
模式验证与断言 是 CoddyKit 上的免费 Pandas & NumPy Academy 课时。 这是第 4 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Pandas & NumPy Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Pandas & NumPy Academy 课程共包含 4 节课。
本课时的部分内容尚未翻译，以英文显示。
Why Schema Validation Matters
A data pipeline processes new data automatically, often without human review. If the schema of the input file changes — a column is renamed, a date format shifts, or a new category appears — the pipeline should fail loudly rather than produce silently wrong output. Schema validation with assertions is the mechanism that enforces contracts between data producers and data consumers, catching problems at the earliest possible moment.
import pandas as pd
import numpy as np
df = pd.read_parquet('sales_treated.parquet')
print('Loaded:', df.shape)
print(df.dtypes)Required Column Checks
The most fundamental validation is checking that all required columns are present. Store the expected column set in a config and assert that it is a subset of the actual columns. This check catches renames and drops immediately at pipeline start, before any downstream code attempts to access missing columns and raises a confusing KeyError deep in the pipeline.
REQUIRED_COLUMNS = {'order_id', 'order_date', 'customer_id',
'product', 'category', 'quantity', 'unit_price', 'revenue'}
missing = REQUIRED_COLUMNS - set(df.columns)
assert not missing, f'Missing required columns: {missing}'
print('All required columns present.')Column Data Type Assertions
After required columns, validate their data types. A date column loaded as object means pd.to_datetime() was not called. An ID column stored as float instead of int64 often means there are NaN values that prevented integer storage. Write type assertions using assert df['col'].dtype == expected_type for the most critical columns.
assert pd.api.types.is_datetime64_any_dtype(df['order_date']), \
'order_date must be datetime'
assert pd.api.types.is_numeric_dtype(df['revenue']), \
'revenue must be numeric'
assert df['order_id'].dtype == object or pd.api.types.is_integer_dtype(df['order_id']), \
'order_id must be string or int'
print('Dtype checks passed.')Non-Null Constraints
Key columns like order_id, order_date, and revenue should never be null. Assert df['col'].notna().all() for each of them. To make the assertion message actionable, include the count of null values so the pipeline operator knows the scale of the problem rather than just that an assertion failed.
NOT_NULL_COLS = ['order_id', 'order_date', 'revenue', 'customer_id']
for col in NOT_NULL_COLS:
null_count = df[col].isna().sum()
assert null_count == 0, f'{col} has {null_count} null values'
print('Non-null checks passed.')Range Checks for Numeric Columns
Numeric columns often have valid business ranges. Revenue must be non-negative. Quantity must be a positive integer. Unit price must be greater than zero. Assert these constraints explicitly — a negative revenue row that slips through will undercount totals in every downstream aggregation without any error. Range checks catch data entry errors and upstream system bugs early.
assert (df['revenue'] >= 0).all(), 'Negative revenue found'
assert (df['quantity'] > 0).all() or df['is_return'].any(), \
'Non-positive quantity without return flag'
assert (df['unit_price'] > 0).all(), 'Zero or negative unit price found'
print('Range checks passed.')Unique Key Assertions
After deduplication, order_id should be unique. Assert df['order_id'].is_unique to verify this invariant is maintained across every pipeline run. Uniqueness violations after deduplication indicate a bug in the deduplication logic or a newly introduced data source that was not cleaned before merging into the main dataset.
assert df['order_id'].is_unique, \
f'order_id not unique: {df.duplicated(subset=["order_id"]).sum()} duplicates'
print('Uniqueness check passed.')Categorical Value Assertions
For columns with a finite set of valid values — like region or category — assert that every value is in the allowed set. This catches rogue values that appear after a system migration or an upstream data entry change. Define the valid sets in your pipeline config so they are easy to update when the business expands into new regions.
VALID_REGIONS = {'North', 'South', 'East', 'West', 'Central'}
VALID_CATEGORIES = {'electronics', 'apparel', 'home', 'sports', 'beauty', 'other'}
assert df['region'].isin(VALID_REGIONS).all(), \
f'Invalid regions: {df[~df["region"].isin(VALID_REGIONS)]["region"].unique()}'
assert df['category'].isin(VALID_CATEGORIES).all(), \
'Invalid categories found'
print('Categorical checks passed.')Date Range Assertions
Validate that all dates fall within the expected period for the dataset. An order dated in the future is impossible; an order dated before the company was founded indicates a corrupted record. Define MIN_DATE and MAX_DATE in the config and assert the column falls within bounds. This also catches Unix-epoch-zero dates (1970-01-01) from bad timestamp conversions.
MIN_DATE = pd.Timestamp('2020-01-01')
MAX_DATE = pd.Timestamp('today')
assert (df['order_date'] >= MIN_DATE).all(), 'Date before minimum found'
assert (df['order_date'] <= MAX_DATE).all(), 'Future date found'
print(f'Date range: {df["order_date"].min()} to {df["order_date"].max()}')Row Count Guard
A sudden change in row count from the previous pipeline run is a strong signal of an upstream issue. Store the previous run's row count in a config file and assert that the new count is within a tolerance band — for example, within ±20 % of the historical count. A dataset that loses 50 % of its rows between runs almost certainly indicates a broken data export.
EXPECTED_MIN_ROWS = 5000
EXPECTED_MAX_ROWS = 200000
assert EXPECTED_MIN_ROWS <= len(df) <= EXPECTED_MAX_ROWS, \
f'Row count {len(df)} outside expected range [{EXPECTED_MIN_ROWS}, {EXPECTED_MAX_ROWS}]'
print(f'Row count check passed: {len(df)} rows')Packaging Checks into a Validate Function
Collect all assertions into a single validate(df) function that can be called at the start of every pipeline stage. Each check raises an AssertionError with a descriptive message if it fails. This pattern makes the pipeline self-documenting: the validation function is the machine-readable schema contract for the DataFrame at that stage.
def validate_sales_df(df):
assert not (REQUIRED_COLUMNS - set(df.columns)), 'Missing columns'
assert df['order_id'].is_unique, 'Duplicate order IDs'
assert (df['revenue'] >= 0).all(), 'Negative revenue'
assert df['region'].isin(VALID_REGIONS).all(), 'Invalid region'
print(f'Validation passed: {len(df)} rows, {len(df.columns)} columns')
validate_sales_df(df)Logging Validation Failures Gracefully
In production pipelines, a hard assert crash is acceptable during development but undesirable in a scheduled job. Replace bare assertions with try/except AssertionError blocks that log the error message and optionally send an alert before exiting. This lets the monitoring system capture the failure reason rather than just an unhandled exception traceback.
import logging
logging.basicConfig(level=logging.INFO)
def validate_with_logging(df):
checks = [
(lambda d: d['order_id'].is_unique, 'Duplicate order IDs'),
(lambda d: (d['revenue'] >= 0).all(), 'Negative revenue found'),
]
for check_fn, msg in checks:
try:
assert check_fn(df), msg
except AssertionError as e:
logging.error(f'VALIDATION FAILED: {e}')
raise
validate_with_logging(df)
print('All checks passed.')Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: asserting required columns, dtypes, non-null constraints, and range validity, checking unique keys, categorical values, and date ranges, and packaging all checks into a reusable validate() function with logging. Next up we explore custom aggregations using apply() on columns and rows.
常见问题解答
「模式验证与断言」课时是免费的吗？
是的 — 「模式验证与断言」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Pandas & NumPy Academy 课程的其余内容，请升级到 CoddyKit PRO。 Pandas & NumPy Academy 课程共包含 4 节课。
「模式验证与断言」这节课中我会学到什么？
针对列范围、非空约束和唯一键编写断言检查，并让它们在每个 pipeline 开始时运行。 你通过在浏览器中直接运行的动手代码来练习 Pandas & NumPy Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。
学习 Pandas & NumPy Academy 需要有经验吗？
无需任何先前经验。CoddyKit 上的 Pandas & NumPy Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 4 节课，共 4 节。
「模式验证与断言」课时需要多长时间？
大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。
我能在这节 Pandas & NumPy Academy 课中编写并运行代码吗？
能。每节 Pandas & NumPy Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。