1. Making Sense of Data with Descriptive Statistics
Descriptive statistics summarize structure and check that values make sense before modeling. Useful tools include:
df.describe() — multi-row summary across numeric features
value_counts() — frequency of distinct values
quantile() — split distributions into probability portions
skew() — whether the curve leans left or right
Core ideas: central tendency (mean, median), dispersion (std, quartiles, IQR), shape (skewness), and simple validity checks (e.g. Q3 > Q1).
2. Quantiles and the Interquartile Range (IQR)
Quartiles each cover about 25% of observations:
- Q1 (25%) — lower quarter boundary
- Q2 (50% / median) — middle of the data
- Q3 (75%) — upper quarter boundary
IQR = Q3 − Q1 measures spread of the middle 50% of values.
3. Skewness
- Right-skewed (positive) — long right tail; mean > median;
.skew() > 0
- Left-skewed (negative) — long left tail; mean < median;
.skew() < 0
4. Practical Example: E-Commerce Revenue
import pandas as pd
df = pd.read_csv("ecommerce_orders.csv", parse_dates=["Order Date"])
print("Dataset Shape:", df.shape)
stats = df[["Unit Price", "Revenue"]].describe()
print(stats)
q1 = df["Revenue"].quantile(0.25)
q2 = df["Revenue"].quantile(0.50)
q3 = df["Revenue"].quantile(0.75)
print(f"Q1: {q1}, Median: {q2}, Q3: {q3}")
print("Mean:", df["Revenue"].mean())
print("Median:", df["Revenue"].median())
print("Skew:", df["Revenue"].skew())
print(df.groupby("Product Category")["Revenue"].mean())
iqr = q3 - q1
print("IQR:", iqr)
print("Q3 > Q1:", q3 > q1)
print("IQR > 0:", iqr > 0)
Illustrative results on a 5,000-row sample: Q1 ≈ 354, median ≈ 796, Q3 ≈ 1,515; mean ≈ 1,021 (above median → right skew ≈ 1.017); IQR ≈ 1,161.
5. Group Means and Validation
groupby(...).mean() compares average revenue by product category. Checking Q3 > Q1 and IQR > 0 confirms the quartile order is sensible.