Validação de esquema e asserções
Escreva verificações por asserção para intervalos de colunas, restrições de valores não nulos e chaves únicas, executadas no início de cada pipeline.
Validação de esquema e asserções é uma aula grátis de Pandas & NumPy Academy no CoddyKit. Esta é a aula 4 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Pandas & NumPy Academy, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Pandas & NumPy Academy inclui 4 aulas no total.
Partes desta aula ainda não foram traduzidas e aparecem em inglês.
Why Schema Validation Matters
A data pipeline processes new data automatically, often without human review. If the schema of the input file changes — a column is renamed, a date format shifts, or a new category appears — the pipeline should fail loudly rather than produce silently wrong output. Schema validation with assertions is the mechanism that enforces contracts between data producers and data consumers, catching problems at the earliest possible moment.
import pandas as pd
import numpy as np
df = pd.read_parquet('sales_treated.parquet')
print('Loaded:', df.shape)
print(df.dtypes)Required Column Checks
The most fundamental validation is checking that all required columns are present. Store the expected column set in a config and assert that it is a subset of the actual columns. This check catches renames and drops immediately at pipeline start, before any downstream code attempts to access missing columns and raises a confusing KeyError deep in the pipeline.
REQUIRED_COLUMNS = {'order_id', 'order_date', 'customer_id',
'product', 'category', 'quantity', 'unit_price', 'revenue'}
missing = REQUIRED_COLUMNS - set(df.columns)
assert not missing, f'Missing required columns: {missing}'
print('All required columns present.')Column Data Type Assertions
After required columns, validate their data types. A date column loaded as object means pd.to_datetime() was not called. An ID column stored as float instead of int64 often means there are NaN values that prevented integer storage. Write type assertions using assert df['col'].dtype == expected_type for the most critical columns.
assert pd.api.types.is_datetime64_any_dtype(df['order_date']), \
'order_date must be datetime'
assert pd.api.types.is_numeric_dtype(df['revenue']), \
'revenue must be numeric'
assert df['order_id'].dtype == object or pd.api.types.is_integer_dtype(df['order_id']), \
'order_id must be string or int'
print('Dtype checks passed.')Non-Null Constraints
Key columns like order_id, order_date, and revenue should never be null. Assert df['col'].notna().all() for each of them. To make the assertion message actionable, include the count of null values so the pipeline operator knows the scale of the problem rather than just that an assertion failed.
NOT_NULL_COLS = ['order_id', 'order_date', 'revenue', 'customer_id']
for col in NOT_NULL_COLS:
null_count = df[col].isna().sum()
assert null_count == 0, f'{col} has {null_count} null values'
print('Non-null checks passed.')Range Checks for Numeric Columns
Numeric columns often have valid business ranges. Revenue must be non-negative. Quantity must be a positive integer. Unit price must be greater than zero. Assert these constraints explicitly — a negative revenue row that slips through will undercount totals in every downstream aggregation without any error. Range checks catch data entry errors and upstream system bugs early.
assert (df['revenue'] >= 0).all(), 'Negative revenue found'
assert (df['quantity'] > 0).all() or df['is_return'].any(), \
'Non-positive quantity without return flag'
assert (df['unit_price'] > 0).all(), 'Zero or negative unit price found'
print('Range checks passed.')Unique Key Assertions
After deduplication, order_id should be unique. Assert df['order_id'].is_unique to verify this invariant is maintained across every pipeline run. Uniqueness violations after deduplication indicate a bug in the deduplication logic or a newly introduced data source that was not cleaned before merging into the main dataset.
assert df['order_id'].is_unique, \
f'order_id not unique: {df.duplicated(subset=["order_id"]).sum()} duplicates'
print('Uniqueness check passed.')Categorical Value Assertions
For columns with a finite set of valid values — like region or category — assert that every value is in the allowed set. This catches rogue values that appear after a system migration or an upstream data entry change. Define the valid sets in your pipeline config so they are easy to update when the business expands into new regions.
VALID_REGIONS = {'North', 'South', 'East', 'West', 'Central'}
VALID_CATEGORIES = {'electronics', 'apparel', 'home', 'sports', 'beauty', 'other'}
assert df['region'].isin(VALID_REGIONS).all(), \
f'Invalid regions: {df[~df["region"].isin(VALID_REGIONS)]["region"].unique()}'
assert df['category'].isin(VALID_CATEGORIES).all(), \
'Invalid categories found'
print('Categorical checks passed.')Date Range Assertions
Validate that all dates fall within the expected period for the dataset. An order dated in the future is impossible; an order dated before the company was founded indicates a corrupted record. Define MIN_DATE and MAX_DATE in the config and assert the column falls within bounds. This also catches Unix-epoch-zero dates (1970-01-01) from bad timestamp conversions.
MIN_DATE = pd.Timestamp('2020-01-01')
MAX_DATE = pd.Timestamp('today')
assert (df['order_date'] >= MIN_DATE).all(), 'Date before minimum found'
assert (df['order_date'] <= MAX_DATE).all(), 'Future date found'
print(f'Date range: {df["order_date"].min()} to {df["order_date"].max()}')Row Count Guard
A sudden change in row count from the previous pipeline run is a strong signal of an upstream issue. Store the previous run's row count in a config file and assert that the new count is within a tolerance band — for example, within ±20 % of the historical count. A dataset that loses 50 % of its rows between runs almost certainly indicates a broken data export.
EXPECTED_MIN_ROWS = 5000
EXPECTED_MAX_ROWS = 200000
assert EXPECTED_MIN_ROWS <= len(df) <= EXPECTED_MAX_ROWS, \
f'Row count {len(df)} outside expected range [{EXPECTED_MIN_ROWS}, {EXPECTED_MAX_ROWS}]'
print(f'Row count check passed: {len(df)} rows')Packaging Checks into a Validate Function
Collect all assertions into a single validate(df) function that can be called at the start of every pipeline stage. Each check raises an AssertionError with a descriptive message if it fails. This pattern makes the pipeline self-documenting: the validation function is the machine-readable schema contract for the DataFrame at that stage.
def validate_sales_df(df):
assert not (REQUIRED_COLUMNS - set(df.columns)), 'Missing columns'
assert df['order_id'].is_unique, 'Duplicate order IDs'
assert (df['revenue'] >= 0).all(), 'Negative revenue'
assert df['region'].isin(VALID_REGIONS).all(), 'Invalid region'
print(f'Validation passed: {len(df)} rows, {len(df.columns)} columns')
validate_sales_df(df)Logging Validation Failures Gracefully
In production pipelines, a hard assert crash is acceptable during development but undesirable in a scheduled job. Replace bare assertions with try/except AssertionError blocks that log the error message and optionally send an alert before exiting. This lets the monitoring system capture the failure reason rather than just an unhandled exception traceback.
import logging
logging.basicConfig(level=logging.INFO)
def validate_with_logging(df):
checks = [
(lambda d: d['order_id'].is_unique, 'Duplicate order IDs'),
(lambda d: (d['revenue'] >= 0).all(), 'Negative revenue found'),
]
for check_fn, msg in checks:
try:
assert check_fn(df), msg
except AssertionError as e:
logging.error(f'VALIDATION FAILED: {e}')
raise
validate_with_logging(df)
print('All checks passed.')Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: asserting required columns, dtypes, non-null constraints, and range validity, checking unique keys, categorical values, and date ranges, and packaging all checks into a reusable validate() function with logging. Next up we explore custom aggregations using apply() on columns and rows.
Perguntas Frequentes
A aula “Validação de esquema e asserções” é grátis?
Sim — o texto completo de “Validação de esquema e asserções” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Pandas & NumPy Academy, atualize para CoddyKit PRO. O curso de Pandas & NumPy Academy inclui 4 aulas no total.
O que vou aprender em “Validação de esquema e asserções”?
Escreva verificações por asserção para intervalos de colunas, restrições de valores não nulos e chaves únicas, executadas no início de cada pipeline. Você pratica Pandas & NumPy Academy com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.
Preciso ter experiência prévia para começar Pandas & NumPy Academy?
Nenhuma experiência prévia é necessária. Pandas & NumPy Academy no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 4 de 4.
Quanto tempo leva a aula “Validação de esquema e asserções”?
A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.
Posso escrever e executar código nesta aula de Pandas & NumPy Academy?
Sim. Cada aula de Pandas & NumPy Academy inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.
Todas as aulas deste curso
- Detectando e removendo duplicatas
- Detecção e tratamento de valores discrepantes
- Padronizando categorias inconsistentes
- Validação de esquema e asserções