Загрузка и проверка набора данных о продажах

A typical sales dataset contains columns like order_id, order_date, customer_id, product, category, quantity, unit_price, and region. Before any analysis you must load this file and take a first look at its structure. The goal of this first step is to understand what data you have before writing a single formula.

After loading, immediately check df.shape to know the dataset dimensions and df.columns to see all column names. This confirms the file loaded correctly and shows whether any column names have unexpected spaces or capitalisation that need cleaning. A mismatch in expected column count is an early warning sign of a corrupted file.

Use df.dtypes or df.info() to see the inferred data type of every column. Common problems include date columns loaded as object (string) and numeric columns loaded as object because of stray commas or currency symbols. Catching dtype issues early prevents silent errors in arithmetic later.

Run df.isna().sum() to count NaN values per column. Convert to a percentage with df.isna().mean() * 100 to see what fraction of each column is missing. Columns with more than 30–40 % missing usually need a decision: drop the column, impute, or flag as a separate indicator variable.

Duplicate order records inflate revenue totals and distort any per-customer analysis. Use df.duplicated().sum() to count exact duplicates and df.duplicated(subset=['order_id']).sum() to check for repeated order IDs, which should be unique. Identifying duplicates at load time saves hours of debugging later.

A date column loaded as object must be converted with pd.to_datetime() so you can do date arithmetic. Pass format='%Y-%m-%d' if you know the format, or use infer_datetime_format=True for mixed formats. After conversion, extract year and month with the .dt accessor for time-based grouping.

Negative quantity values typically represent returns or data entry errors. Use boolean indexing to find them: df[df['quantity'] < 0]. Decide whether to treat them as legitimate returns (keep with a flag) or as errors (drop or correct). Always document your decision so the pipeline is reproducible.

Use df.describe() to see the min, max, mean, and quartiles of every numeric column. A unit_price with a minimum of zero or a negative value is suspicious. A quantity maximum far above any reasonable order size might be a data entry error. Sanity-checking ranges is a fast way to spot anomalies.

For columns like region and category, use df['region'].value_counts() to see all unique values and their frequencies. Look for typos, inconsistent capitalisation (e.g. 'north' vs 'North'), or unexpected values that indicate upstream data issues. Standardise them before any groupby operation.

A structured audit summary DataFrame helps communicate data quality issues to stakeholders. Build one by collecting metrics like row count, column count, missing counts, and duplicate counts into a dictionary and converting it to a DataFrame. This summary becomes the first section of your analysis report and justifies every cleaning step you take.

After the initial audit and basic fixes (dtype corrections, duplicate removal, flag columns), save a clean snapshot so downstream steps always start from a consistent baseline. Use df.to_csv('sales_clean.csv', index=False) or, for faster reloading, df.to_parquet('sales_clean.parquet'). Parquet preserves dtypes including datetime, which CSV does not.

Test your understanding of Data Analysis concepts from this lesson.

In this lesson you learned: loading a CSV and checking shape/columns, auditing dtypes, missing values, duplicates, and negative quantities, and saving a clean snapshot for downstream steps. Next up we explore revenue calculations and feature engineering on the clean dataset.