Digital Animation and Graphic Design Course for Kids in Sengkang, Singapore

1. Introduction: The Reality of Real-World Data

Client and system data often arrive with missing or null values from recording failures, corruption, or uncollected fields. Models and stats break or give wrong results if nulls are ignored, so finding and fixing missing data is a core preprocessing step.

• Detect: df.info(), df.isna().sum()
• Impute (preferred): numeric → mean/median; categorical → mode
• Drop when needed: df.dropna(subset=[...], inplace=True)

2. Dataset Overview: retail_sales.csv

Six features: Date, Category, Sales, Quantity, Profit, Region. Ideal would be 1,825 fully filled rows; in practice several columns have nulls that must be resolved systematically.

3. Detecting Missing Data in Pandas

A. Non-null counts with df.info()

import pandas as pd

df = pd.read_csv("retail_sales.csv")
df.info()

Example: total 1,825 rows; Sales might show 1,823 non-null (2 missing); Region 1,821 non-null (4 missing).

B. Exact null counts with df.isna().sum()

missing_counts = df.isna().sum()
print("Missing values per column:")
print(missing_counts)

isna() marks nulls as True; .sum() counts them per column. Typical pattern: Date/Category often complete; Sales, Quantity, and Region may have a few nulls.

4. Two Strategies for Handling Missing Values

1. Drop rows that contain nulls
2. Impute (fill) with estimated values — preferred when a reliable fill is possible

Imputation keeps other columns in the same row and preserves training volume; dropping shrinks the dataset.

5. Strategy 1: Imputation with Basic Statistics

• Numeric (Sales, Profit): mean or median — e.g. df['Sales'].fillna(df['Sales'].median())
• Categorical (Category, Region): mode — e.g. df['Category'].fillna(df['Category'].mode()[0])

A. Quantity with mode

df["Quantity"] = pd.to_numeric(df["Quantity"])
quantity_mode = df["Quantity"].mode()[0]
df["Quantity"] = df["Quantity"].fillna(quantity_mode)

.mode() returns a Series; use index 0 for the most frequent value.

B. Sales with median

sales_median = df["Sales"].median()
df["Sales"] = df["Sales"].fillna(sales_median)

Median is robust to outliers in continuous financial columns.

C. Category with mode

category_mode = df["Category"].mode()[0]
df["Category"] = df["Category"].fillna(category_mode)

After filling, df.info() and df.isna().sum() should show full non-null counts for those columns.

6. Strategy 2: Dropping Rows with dropna()

When a field cannot be imputed reliably (e.g. critical Region values), drop those rows:

df.dropna(subset=["Region"], inplace=True)

print("Final Missing Counts Across All Columns:")
print(df.isna().sum())

• subset=['Region'] — only rows with null Region are removed
• inplace=True — updates the DataFrame in place

A final isna().sum() of zeros means the table is ready for EDA and modeling.