Arduino Microcontroller Circuit Design Classes in Songdo, Incheon

1. What Are Outliers and Why Remove Them?

Outliers are extreme values far from the bulk of the data. They can come from entry errors, system bugs, or rare real events. Because the mean and variance are sensitive to extremes, outliers can distort summaries and mislead models. Detection is a standard preprocessing step.

2. Two Statistical Methods (and Domain Check)

1. IQR method — bounds from Q1 and Q3
2. Z-score method — distance from the mean in standard deviations
3. SME review — domain judgment, ideally confirmed with statistics

3. IQR Method

IQR = Q3 − Q1. Bounds: lower = Q1 − 1.5×IQR, upper = Q3 + 1.5×IQR. Points outside are flagged.

import pandas as pd

df = pd.read_csv("ecommerce_orders.csv")

q1 = df["Revenue"].quantile(0.25)
q3 = df["Revenue"].quantile(0.75)
iqr = q3 - q1
lower_bound = q1 - 1.5 * iqr
upper_bound = q3 + 1.5 * iqr

outliers_iqr = df[(df["Revenue"] < lower_bound) | (df["Revenue"] > upper_bound)]
print("Outlier rows:", outliers_iqr.shape)

df_cleaned_iqr = df[(df["Revenue"] >= lower_bound) & (df["Revenue"] <= upper_bound)]
print("Cleaned shape:", df_cleaned_iqr.shape)

On a 5,000-row sample: 67 outliers flagged; ~4,933 rows remain. Strong option for skewed sales/time-series data.

4. Z-Score Method

Z = (x − μ) / σ. Common rule: |Z| > 3 is an outlier.

import numpy as np

mean_rev = df["Revenue"].mean()
std_rev = df["Revenue"].std()
df["z_score"] = (df["Revenue"] - mean_rev) / std_rev

outliers_zscore = df[np.abs(df["z_score"]) > 3]
df_cleaned_zscore = df[np.abs(df["z_score"]) <= 3]
print(outliers_zscore.shape, df_cleaned_zscore.shape)

Sample result: 26 outliers; ~4,974 rows remain. Works best when the feature is roughly symmetric/normal.

5. Comparison

Criterion	IQR	Z-Score
Basis	Quartiles ± 1.5×IQR	|Z| > 3 from mean
Example flags (5k rows)	~67	~26
Best for	Skewed sales / time series	Near-normal features

SME suggestions should still be checked with IQR or Z-score to limit human bias.