Data Cleaning and Preprocessing

Raw data is often messy and requires cleaning and preprocessing before it can be analyzed. These steps are crucial for ensuring the quality and accuracy of your analysis.

In this lesson, you’ll learn techniques to handle missing data, remove duplicates, and preprocess data for analysis.

What Is Data Cleaning?

Data cleaning involves identifying and fixing errors in the dataset. Common tasks include:

• Handling missing values.
• Removing duplicates.
• Standardizing formats.

Handling Missing Data

Missing data can occur due to various reasons. You can handle it by:

• Filling missing values with a default value or mean/median.
• Dropping rows or columns with too many missing values.

# Example: Handling missing data
import pandas as pd

data = {'Name': ['Alice', 'Bob', None], 'Age': [25, None, 30]}
df = pd.DataFrame(data)
df['Age'].fillna(df['Age'].mean(), inplace=True)
print(df)

Removing Duplicates

Duplicate data can skew your analysis. Use Pandas to remove duplicates:

# Example: Removing duplicates
data = {'Name': ['Alice', 'Bob', 'Alice'], 'Age': [25, 30, 25]}
df = pd.DataFrame(data)
df = df.drop_duplicates()
print(df)

Standardizing Data

Data standardization ensures consistency. For example, you can standardize date formats or text case:

# Example: Standardizing text case
data = {'Name': ['Alice', 'BOB', 'alice']}
df = pd.DataFrame(data)
df['Name'] = df['Name'].str.capitalize()
print(df)

Transforming Data

Transformations like scaling and encoding are part of preprocessing:

• Scaling: Standardizing numerical values.
• Encoding: Converting categorical data into numerical form.

# Example: Encoding categorical data
from sklearn.preprocessing import LabelEncoder

data = {'Category': ['A', 'B', 'A']}
df = pd.DataFrame(data)
encoder = LabelEncoder()
df['Category'] = encoder.fit_transform(df['Category'])
print(df)

Scaling Data

Scaling ensures numerical data is on the same scale, which is essential for machine learning algorithms:

# Example: Scaling data
from sklearn.preprocessing import StandardScaler

values = [[10], [20], [30]]
scaler = StandardScaler()
scaled_values = scaler.fit_transform(values)
print(scaled_values)

Validating Data

Validation ensures the data meets quality standards. For example, check for outliers or invalid values:

# Example: Validating data
import numpy as np

data = [10, 20, 1000]  # 1000 might be an outlier
mean = np.mean(data)
std_dev = np.std(data)
outliers = [x for x in data if abs(x - mean) > 2 * std_dev]
print("Outliers:", outliers)

Common Mistakes in Data Cleaning

Here are some mistakes to avoid:

• Ignoring missing data, leading to inaccurate analysis.
• Not standardizing formats, causing inconsistencies.
• Overlooking validation, leading to errors in downstream tasks.

What Did We Learn?

In this lesson, you learned:

• How to handle missing data and remove duplicates.
• How to standardize, transform, and scale data for analysis.
• How to validate data quality and identify outliers.
• The importance of data cleaning for accurate analysis.

Great job! Let’s move to the next topic.