Поиск и удаление дубликатов
Находите полные и частичные дубликаты с помощью duplicated() и drop_duplicates() и решайте, какую дублирующуюся запись сохранить.
«Поиск и удаление дубликатов» — бесплатный урок Pandas & NumPy Academy на CoddyKit. Это урок 1 из 4. Ты можешь прочитать весь урок бесплатно ниже — а потом практиковать его прямо в браузере с встроенным редактором кода и ИИ-репетитором 24/7. Это часть пути обучения Pandas & NumPy Academy, и твой прогресс синхронизируется между веб-версией и приложением CoddyKit. Курс Pandas & NumPy Academy содержит 4 уроков всего.
Части этого урока еще не переведены и отображаются на английском.
Why Duplicates Matter
Duplicate rows silently inflate counts, revenue totals, and averages without any error message. A sales dataset with 500 duplicate order records will overstate revenue by the exact sum of those 500 orders. Detecting and removing duplicates is one of the first cleaning steps you should perform, before any aggregation or modelling, because downstream errors compound from this single source of corruption.
import pandas as pd
df = pd.read_csv('orders.csv')
print('Shape before dedup:', df.shape)
print('Exact duplicates:', df.duplicated().sum())Finding Exact Duplicates
df.duplicated() returns a boolean Series that is True for every row that is an exact copy of a previous row. The default keep='first' marks all but the first occurrence. Passing keep=False marks every occurrence of a duplicate — useful when you want to inspect all copies of a duplicated record before deciding which to keep.
# Mark only the second+ occurrences
partial_dups = df[df.duplicated(keep='first')]
print('Rows to remove:', len(partial_dups))
# Mark ALL copies of any duplicate
all_dups = df[df.duplicated(keep=False)]
print('Rows involved in duplicates:', len(all_dups))Dropping Exact Duplicates
Remove exact duplicate rows with df.drop_duplicates(). By default it keeps the first occurrence and drops all others. Pass keep='last' to keep the most recent record if your data has a timestamp and later rows are considered more authoritative. Always check the row count before and after to confirm the expected number of rows were removed.
df_clean = df.drop_duplicates(keep='first')
print('Rows removed:', len(df) - len(df_clean))
print('Shape after dedup:', df_clean.shape)Partial Duplicates: Key-Based
Rows may be partial duplicates: they share a business key (like order_id) but differ in other columns due to an upstream bug that created two versions of the same record. Use df.duplicated(subset=['order_id']) to find rows with the same key but potentially different values in other columns. Inspect these rows before deciding how to reconcile them.
key_dups = df[df.duplicated(subset=['order_id'], keep=False)]
print('Orders with duplicate IDs:')
print(key_dups.sort_values('order_id').head(10))Choosing Which Duplicate to Keep
When partial duplicates exist, you must decide which record is authoritative. Common strategies: keep the row with the latest timestamp (most recent update), keep the row with the most non-null values, or keep the row with the higher dollar amount if one record has corrections. Sort by your tie-breaking criterion and then drop duplicates keeping the first (or last) occurrence.
# Keep the record with the latest update timestamp
df_sorted = df.sort_values('updated_at', ascending=False)
df_dedup = df_sorted.drop_duplicates(subset=['order_id'], keep='first')
print('Kept:', len(df_dedup), 'unique orders')Near-Duplicate Detection
Near-duplicates are rows that represent the same entity but differ slightly — for example, the same customer name with a typo, or the same transaction with a 1-cent rounding difference. Exact duplicate detection misses these. One approach is to group by the key column and count: if a customer name appears with slight variations, use .str.strip().str.lower() to normalise before deduplication.
df['customer_normalized'] = df['customer_name'].str.strip().str.lower()
near_dups = df.groupby('customer_normalized').size().sort_values(ascending=False)
print(near_dups[near_dups > 1].head())Deduplicating with GroupBy Aggregation
When you need to merge duplicate rows rather than simply drop them, use groupby().agg(). For example, if two rows represent the same order but one has the shipping address and the other has the billing address, you can aggregate with 'first' (non-null preferred) or a custom function that coalesces non-null values across duplicates.
def coalesce(*args):
for a in args:
if pd.notna(a):
return a
return None
df_merged = df.groupby('order_id').agg(
customer=('customer_name', 'first'),
amount=('amount', 'max'),
date=('order_date', 'min')
).reset_index()
print(df_merged.head())Checking for Row-Order Sensitivity
The result of drop_duplicates(keep='first') depends on row order. If the DataFrame was loaded from a database query without an ORDER BY clause, row order is non-deterministic, meaning the kept record may vary between runs. Always sort by a stable key (e.g. primary key or timestamp) before deduplication to make the process deterministic and reproducible.
# Sort by primary key before deduplication for deterministic results
df = df.sort_values('order_id').reset_index(drop=True)
df_clean = df.drop_duplicates(subset=['order_id'], keep='first')
print('Deterministic dedup complete.')Verifying the Deduplication Result
After deduplication, verify the result with three checks: no remaining exact duplicates (df_clean.duplicated().sum() == 0), no duplicate business keys (df_clean.duplicated(subset=['order_id']).sum() == 0), and the expected row count is plausible. Write these as assertions so they fail loudly if future data changes invalidate the cleaning logic.
assert df_clean.duplicated().sum() == 0, 'Exact duplicates remain'
assert df_clean.duplicated(subset=['order_id']).sum() == 0, 'Duplicate order IDs remain'
print('Deduplication verified. Rows:', len(df_clean))Documenting Duplicates Removed
Good data cleaning is reproducible and auditable. Log the number of rows removed, the deduplication key, and the tie-breaking rule into a cleaning report dictionary. Persist this report alongside the clean data file so anyone running the pipeline later can verify the cleaning decisions without re-reading the code. Transparency in data cleaning builds trust in analysis results.
cleaning_log = {
'original_rows': len(df),
'dedup_key': 'order_id',
'tiebreak': 'keep first by updated_at desc',
'rows_removed': len(df) - len(df_clean),
'clean_rows': len(df_clean)
}
for k, v in cleaning_log.items():
print(f'{k}: {v}')Saving the Deduplicated Dataset
Save the deduplicated DataFrame to a new file rather than overwriting the original. This preserves the raw data for auditing. Name the file clearly with a _clean or _deduped suffix and include the run date so multiple cleaning passes are identifiable. Use Parquet for fast reloading in subsequent pipeline steps.
from datetime import date
output_path = f'orders_clean_{date.today()}.parquet'
df_clean.to_parquet(output_path, index=False)
print(f'Saved {len(df_clean)} rows to {output_path}')Quick Check
Test your understanding of Data Analysis concepts from this lesson.
Lesson Recap
In this lesson you learned: detecting exact and partial duplicates with duplicated() and drop_duplicates(), choosing which duplicate to keep using sorting and aggregation strategies, and verifying results with assertions and logging the cleaning decisions. Next up we explore outlier detection and treatment techniques.
Часто задаваемые вопросы
Урок «Поиск и удаление дубликатов» бесплатный?
Да — полный текст урока «Поиск и удаление дубликатов» бесплатно доступен здесь в веб-версии. Чтобы практиковать его интерактивно (встроенный редактор кода и ИИ-репетитор 24/7) и разблокировать остальной курс Pandas & NumPy Academy, подпишись на CoddyKit PRO. Курс Pandas & NumPy Academy содержит 4 уроков всего.
Чему я научусь в уроке «Поиск и удаление дубликатов»?
Находите полные и частичные дубликаты с помощью duplicated() и drop_duplicates() и решайте, какую дублирующуюся запись сохранить. Ты практикуешь Pandas & NumPy Academy с помощью реального кода, который запускаешь прямо в браузере, и ИИ-репетитор 24/7 отвечает на твои вопросы во время урока.
Нужен ли мне опыт, чтобы начать Pandas & NumPy Academy?
Предыдущий опыт не требуется. Pandas & NumPy Academy на CoddyKit структурирован для всех уровней — от новичков до продвинутых, поэтому ты можешь начать отсюда или с самого начала и учиться в своем темпе. Это урок 1 из 4.
Сколько времени занимает урок «Поиск и удаление дубликатов»?
Большинство уроков CoddyKit занимают около 5–10 минут. Каждый из них компактный и интерактивный, поэтому ты постоянно делаешь прогресс и продолжаешь с того же места в веб-версии и приложении.
Можно ли писать и запускать код в этом уроке Pandas & NumPy Academy?
Да. Каждый урок Pandas & NumPy Academy включает встроенный редактор кода, поэтому ты пишешь и запускаешь реальный код прямо в браузере и получаешь моментальную обратную связь от AI — локальная установка не требуется.
Все уроки этого курса
- Поиск и удаление дубликатов
- Поиск и обработка выбросов
- Приведение несогласованных категорий к единому виду
- Проверка схемы и утверждения