0Pricing
Pandas & NumPy Academy · Aula

Detectando e removendo duplicatas

Encontre duplicatas exatas e parciais com duplicated() e drop_duplicates() e decida qual registro duplicado manter.

Detectando e removendo duplicatas é uma aula grátis de Pandas & NumPy Academy no CoddyKit. Esta é a aula 1 de 4. Você pode ler a aula completa abaixo gratuitamente — depois pratica ao vivo no navegador com um editor de código integrado e um tutor de IA 24/7. Faz parte do caminho de aprendizado de Pandas & NumPy Academy, e seu progresso é sincronizado entre a web e o app CoddyKit. O curso de Pandas & NumPy Academy inclui 4 aulas no total.

Partes desta aula ainda não foram traduzidas e aparecem em inglês.

Why Duplicates Matter

Duplicate rows silently inflate counts, revenue totals, and averages without any error message. A sales dataset with 500 duplicate order records will overstate revenue by the exact sum of those 500 orders. Detecting and removing duplicates is one of the first cleaning steps you should perform, before any aggregation or modelling, because downstream errors compound from this single source of corruption.

import pandas as pd

df = pd.read_csv('orders.csv')
print('Shape before dedup:', df.shape)
print('Exact duplicates:', df.duplicated().sum())

Finding Exact Duplicates

df.duplicated() returns a boolean Series that is True for every row that is an exact copy of a previous row. The default keep='first' marks all but the first occurrence. Passing keep=False marks every occurrence of a duplicate — useful when you want to inspect all copies of a duplicated record before deciding which to keep.

# Mark only the second+ occurrences
partial_dups = df[df.duplicated(keep='first')]
print('Rows to remove:', len(partial_dups))

# Mark ALL copies of any duplicate
all_dups = df[df.duplicated(keep=False)]
print('Rows involved in duplicates:', len(all_dups))

Dropping Exact Duplicates

Remove exact duplicate rows with df.drop_duplicates(). By default it keeps the first occurrence and drops all others. Pass keep='last' to keep the most recent record if your data has a timestamp and later rows are considered more authoritative. Always check the row count before and after to confirm the expected number of rows were removed.

df_clean = df.drop_duplicates(keep='first')

print('Rows removed:', len(df) - len(df_clean))
print('Shape after dedup:', df_clean.shape)

Partial Duplicates: Key-Based

Rows may be partial duplicates: they share a business key (like order_id) but differ in other columns due to an upstream bug that created two versions of the same record. Use df.duplicated(subset=['order_id']) to find rows with the same key but potentially different values in other columns. Inspect these rows before deciding how to reconcile them.

key_dups = df[df.duplicated(subset=['order_id'], keep=False)]
print('Orders with duplicate IDs:')
print(key_dups.sort_values('order_id').head(10))

Choosing Which Duplicate to Keep

When partial duplicates exist, you must decide which record is authoritative. Common strategies: keep the row with the latest timestamp (most recent update), keep the row with the most non-null values, or keep the row with the higher dollar amount if one record has corrections. Sort by your tie-breaking criterion and then drop duplicates keeping the first (or last) occurrence.

# Keep the record with the latest update timestamp
df_sorted = df.sort_values('updated_at', ascending=False)
df_dedup = df_sorted.drop_duplicates(subset=['order_id'], keep='first')

print('Kept:', len(df_dedup), 'unique orders')

Near-Duplicate Detection

Near-duplicates are rows that represent the same entity but differ slightly — for example, the same customer name with a typo, or the same transaction with a 1-cent rounding difference. Exact duplicate detection misses these. One approach is to group by the key column and count: if a customer name appears with slight variations, use .str.strip().str.lower() to normalise before deduplication.

df['customer_normalized'] = df['customer_name'].str.strip().str.lower()

near_dups = df.groupby('customer_normalized').size().sort_values(ascending=False)
print(near_dups[near_dups > 1].head())

Deduplicating with GroupBy Aggregation

When you need to merge duplicate rows rather than simply drop them, use groupby().agg(). For example, if two rows represent the same order but one has the shipping address and the other has the billing address, you can aggregate with 'first' (non-null preferred) or a custom function that coalesces non-null values across duplicates.

def coalesce(*args):
    for a in args:
        if pd.notna(a):
            return a
    return None

df_merged = df.groupby('order_id').agg(
    customer=('customer_name', 'first'),
    amount=('amount', 'max'),
    date=('order_date', 'min')
).reset_index()
print(df_merged.head())

Checking for Row-Order Sensitivity

The result of drop_duplicates(keep='first') depends on row order. If the DataFrame was loaded from a database query without an ORDER BY clause, row order is non-deterministic, meaning the kept record may vary between runs. Always sort by a stable key (e.g. primary key or timestamp) before deduplication to make the process deterministic and reproducible.

# Sort by primary key before deduplication for deterministic results
df = df.sort_values('order_id').reset_index(drop=True)
df_clean = df.drop_duplicates(subset=['order_id'], keep='first')

print('Deterministic dedup complete.')

Verifying the Deduplication Result

After deduplication, verify the result with three checks: no remaining exact duplicates (df_clean.duplicated().sum() == 0), no duplicate business keys (df_clean.duplicated(subset=['order_id']).sum() == 0), and the expected row count is plausible. Write these as assertions so they fail loudly if future data changes invalidate the cleaning logic.

assert df_clean.duplicated().sum() == 0, 'Exact duplicates remain'
assert df_clean.duplicated(subset=['order_id']).sum() == 0, 'Duplicate order IDs remain'
print('Deduplication verified. Rows:', len(df_clean))

Documenting Duplicates Removed

Good data cleaning is reproducible and auditable. Log the number of rows removed, the deduplication key, and the tie-breaking rule into a cleaning report dictionary. Persist this report alongside the clean data file so anyone running the pipeline later can verify the cleaning decisions without re-reading the code. Transparency in data cleaning builds trust in analysis results.

cleaning_log = {
    'original_rows': len(df),
    'dedup_key': 'order_id',
    'tiebreak': 'keep first by updated_at desc',
    'rows_removed': len(df) - len(df_clean),
    'clean_rows': len(df_clean)
}
for k, v in cleaning_log.items():
    print(f'{k}: {v}')

Saving the Deduplicated Dataset

Save the deduplicated DataFrame to a new file rather than overwriting the original. This preserves the raw data for auditing. Name the file clearly with a _clean or _deduped suffix and include the run date so multiple cleaning passes are identifiable. Use Parquet for fast reloading in subsequent pipeline steps.

from datetime import date

output_path = f'orders_clean_{date.today()}.parquet'
df_clean.to_parquet(output_path, index=False)
print(f'Saved {len(df_clean)} rows to {output_path}')

Quick Check

Test your understanding of Data Analysis concepts from this lesson.

Lesson Recap

In this lesson you learned: detecting exact and partial duplicates with duplicated() and drop_duplicates(), choosing which duplicate to keep using sorting and aggregation strategies, and verifying results with assertions and logging the cleaning decisions. Next up we explore outlier detection and treatment techniques.

Perguntas Frequentes

A aula “Detectando e removendo duplicatas” é grátis?

Sim — o texto completo de “Detectando e removendo duplicatas” é grátis para ler aqui na web. Para praticá-la interativamente (um editor de código integrado e um tutor de IA 24/7) e desbloquear o restante do curso de Pandas & NumPy Academy, atualize para CoddyKit PRO. O curso de Pandas & NumPy Academy inclui 4 aulas no total.

O que vou aprender em “Detectando e removendo duplicatas”?

Encontre duplicatas exatas e parciais com duplicated() e drop_duplicates() e decida qual registro duplicado manter. Você pratica Pandas & NumPy Academy com código prático que executa diretamente no navegador, e um tutor de IA 24/7 responde suas dúvidas enquanto trabalha na aula.

Preciso ter experiência prévia para começar Pandas & NumPy Academy?

Nenhuma experiência prévia é necessária. Pandas & NumPy Academy no CoddyKit é estruturado para alunos iniciantes até avançados, então você pode começar aqui ou desde o início e aprender no seu ritmo. Esta é a aula 1 de 4.

Quanto tempo leva a aula “Detectando e removendo duplicatas”?

A maioria das aulas CoddyKit leva cerca de 5–10 minutos. Cada uma é compacta e interativa, então você faz progresso constante e retoma exatamente de onde parou entre web e app.

Posso escrever e executar código nesta aula de Pandas & NumPy Academy?

Sim. Cada aula de Pandas & NumPy Academy inclui um editor de código integrado, então você escreve e executa código real direto no navegador e recebe feedback de IA instantaneamente — nenhuma configuração local necessária.

Todas as aulas deste curso

  1. Detectando e removendo duplicatas
  2. Detecção e tratamento de valores discrepantes
  3. Padronizando categorias inconsistentes
  4. Validação de esquema e asserções
← Voltar para Pandas & NumPy Academy