Computational Thinking and Logic Building Classes in Novena, Singapore

1. Introduction: Why Data Type Conversion Matters in Pandas

Raw files rarely arrive with perfect data types. Numbers may be read as text, dates as plain objects, and repeating categories duplicated across thousands of rows. If types are not fixed:

• Math fails on numbers stored as strings
• Date work is hard without a proper datetime type
• Memory balloons when repeated text is stored as object

Correct types simplify analysis, speed computation, and prepare data for machine learning.

• Dates: parse_dates=['Date'] → datetime
• Numeric strings: pd.to_numeric(col) → int64 / float64
• Repeating groups: col.astype('category') → integer codes (less RAM)

2. The Case Study Dataset: retail_sales.csv

Example: a nationwide retail file with 1,825 rows and six columns:

1. Date — transaction date
2. Category — product line
3. Sales — total sales
4. Quantity — items sold
5. Profit — profit
6. Region — store location

Goal: load the file, fix types, and optimize memory.

3. Parsing Dates During File Import (parse_dates)

import pandas as pd

# Reading retail_sales.csv and parsing the Date column as datetime
df = pd.read_csv("retail_sales.csv", parse_dates=["Date"])

Without this, dates stay as text. With parse_dates, you can extract years, months, quarters, and run time-series logic without manual string parsing.

4. Converting Numbers Stored as Text (pd.to_numeric)

Formatting quirks can leave numeric columns as object text, so sums and means fail. Fix with pd.to_numeric():

# Converting the 'Profit' column into a numeric format
df["Profit"] = pd.to_numeric(df["Profit"])

# Verifying the converted data type
print("Profit dtype:", df["Profit"].dtype)

After conversion (typically float64), averages, totals, and margins work normally.

5. Converting Text to Categorical Types

Columns like Category and Region repeat a small set of labels. Storing each row as a full string wastes memory. The category type keeps a dictionary of unique labels and stores a small integer code per row.

A. On import (dtype)

df = pd.read_csv(
    "retail_sales.csv",
    parse_dates=["Date"],
    dtype={"Category": "category"}
)

B. After import (.astype)

print("Before conversion:", df["Region"].dtype)
df["Region"] = df["Region"].astype("category")
print("After conversion: ", df["Region"].dtype)

6. Memory Optimization Benchmark

# Inspecting the DataFrame's overall memory footprint
df.info()

• Before: unoptimized text columns can use on the order of ~73.4 KB for this sample
• After converting Region and Category to category: footprint can drop to ~23.4 KB

Converting repeating text columns to category is a standard Pandas practice for large tables.