0Pricing
Pandas & NumPy Academy · Lesson

Efficient Data Types for Memory Reduction

Downcast numeric columns to int32/float32 and convert string columns to Categorical to cut DataFrame memory by up to 70%.

Efficient Data Types for Memory Reduction is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Data Types Affect Performance

In Pandas, every column has a dtype (data type) that determines how values are stored in memory and how fast operations run. Pandas uses wide types by default when loading data: int64 (8 bytes per value), float64 (8 bytes), and object (variable, often 50-200 bytes per string). For millions of rows, choosing smaller types can reduce memory by 50-80% and speed up operations by 2-5x due to better cache utilisation.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'id': np.arange(1000000, dtype='int64'),
    'score': np.random.uniform(0, 100, 1000000).astype('float64'),
    'category': np.random.choice(['A','B','C','D'], 1000000)
})

mem = df.memory_usage(deep=True)
print('Memory per column:')
print(mem)
print(f'Total: {mem.sum() / 1e6:.1f} MB')

Downcasting Integer Columns

If a column contains integer values that fit in a smaller range, downcast it from int64 to a smaller integer type. pd.to_numeric(series, downcast='integer') automatically selects the smallest integer type that can hold all values: int8 (–128 to 127, 1 byte), int16 (–32768 to 32767, 2 bytes), int32 (±2 billion, 4 bytes), or stays as int64 if needed. An int8 column uses 8x less memory than int64.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'age': np.random.randint(18, 90, 500000).astype('int64'),
    'score': np.random.randint(0, 100, 500000).astype('int64'),
    'large_id': np.random.randint(0, 2**31, 500000).astype('int64')
})

# Downcast integers
for col in df.select_dtypes('int64').columns:
    df[col] = pd.to_numeric(df[col], downcast='integer')

print('Dtypes after downcasting:')
print(df.dtypes)
print(f'\nMemory: {df.memory_usage(deep=True).sum()/1e6:.2f} MB')

Downcasting Float Columns

pd.to_numeric(series, downcast='float') converts float64 to float32 (4 bytes instead of 8 bytes) wherever precision allows. float32 has about 7 decimal digits of precision vs. 15 for float64. For most analytics tasks (percentages, prices, normalised features), float32 precision is sufficient. For scientific computing requiring high precision, stay with float64. Halving float precision halves memory and improves cache performance.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'lat': np.random.uniform(-90, 90, 1000000),
                   'lon': np.random.uniform(-180, 180, 1000000),
                   'temp': np.random.uniform(-40, 50, 1000000)})

print('Before:', df.dtypes.unique())
print(f'Memory: {df.memory_usage(deep=True).sum()/1e6:.1f} MB')

for col in df.select_dtypes('float64').columns:
    df[col] = pd.to_numeric(df[col], downcast='float')

print('After:', df.dtypes.unique())
print(f'Memory: {df.memory_usage(deep=True).sum()/1e6:.1f} MB')

The Categorical Data Type

The Categorical dtype is the single biggest memory win for string columns with repeated values. Instead of storing the same string (e.g. 'Electronics') thousands of times, Pandas stores a dictionary of unique values plus a compact integer code for each row. A column with 1 million rows and only 50 unique categories goes from ~50 MB (object dtype) to ~1 MB (Categorical) — a 50x reduction in memory.

import pandas as pd
import numpy as np

np.random.seed(0)
categories = ['Electronics', 'Clothing', 'Books', 'Food', 'Furniture',
              'Sports', 'Toys', 'Health', 'Garden', 'Automotive']

df = pd.DataFrame({'product_cat': np.random.choice(categories, 1000000)})

mem_before = df.memory_usage(deep=True).sum()
df['product_cat'] = df['product_cat'].astype('category')
mem_after = df.memory_usage(deep=True).sum()

print(f'Object dtype: {mem_before/1e6:.1f} MB')
print(f'Categorical:  {mem_after/1e6:.2f} MB')
print(f'Reduction: {mem_before/mem_after:.0f}x')

When Categorical Saves Memory

Categorical dtype saves memory only when the column has many fewer unique values than total rows. The break-even point is when the ratio of unique values to total rows (cardinality) is above about 50%: if every row has a unique string, Categorical actually uses more memory than object dtype because it stores both the code array and the categories array. Always check df['col'].nunique() / len(df) before converting — a ratio below 0.5 (and ideally below 0.05) means Categorical will help.

import pandas as pd
import numpy as np

def memory_gain(df, col):
    n = len(df)
    n_unique = df[col].nunique()
    ratio = n_unique / n
    mem_obj = df[col].memory_usage(deep=True)
    df2 = df.copy()
    df2[col] = df2[col].astype('category')
    mem_cat = df2[col].memory_usage(deep=True)
    print(f'{col}: {n_unique} unique / {n} total (ratio={ratio:.3f})')
    print(f'  Object: {mem_obj/1e6:.2f} MB → Categorical: {mem_cat/1e6:.2f} MB')
    print(f'  {"Saves" if mem_cat < mem_obj else "Wastes"} {abs(mem_obj-mem_cat)/1e6:.2f} MB')

np.random.seed(0)
df = pd.DataFrame({
    'low_card': np.random.choice(['A','B','C'], 500000),   # low cardinality
    'high_card': [f'id_{i}' for i in range(500000)]         # high cardinality
})
memory_gain(df, 'low_card')
memory_gain(df, 'high_card')

Categorical Boosts GroupBy Performance

Beyond memory savings, Categorical dtype also speeds up groupby operations because Pandas can use the integer codes directly rather than hashing strings to find group boundaries. For DataFrames with millions of rows grouped by low-cardinality string columns (like region, product category, or status), converting those columns to Categorical before groupby can yield 2-5x speedups.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({
    'region': np.random.choice(['North','South','East','West'], 1000000),
    'sales': np.random.randn(1000000)
})

# Object dtype groupby
t1 = timeit.timeit(lambda: df.groupby('region')['sales'].mean(), number=50)

# Categorical dtype groupby
df2 = df.copy()
df2['region'] = df2['region'].astype('category')
t2 = timeit.timeit(lambda: df2.groupby('region')['sales'].mean(), number=50)

print(f'Object dtype groupby: {t1/50*1000:.2f} ms')
print(f'Categorical groupby:  {t2/50*1000:.2f} ms')
print(f'Speedup: {t1/t2:.1f}x')

Using boolean dtype for Flag Columns

Binary columns (True/False, 0/1, yes/no) are often stored as object or int64. Converting them to bool dtype uses only 1 byte per value (vs. 8 bytes for int64 or 50+ bytes for string 'yes'/'no'). Use astype(bool) after ensuring the column only contains 0/1 or True/False values. Boolean columns also enable faster filtering because Pandas can use bitwise operations internally.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'is_premium': np.random.choice([0, 1], 1000000).astype('int64'),
    'has_discount': np.random.choice(['yes', 'no'], 1000000)
})

print('Before:')
print(df.dtypes)
print(f'Memory: {df.memory_usage(deep=True).sum()/1e6:.1f} MB')

df['is_premium'] = df['is_premium'].astype(bool)
df['has_discount'] = df['has_discount'].map({'yes': True, 'no': False}).astype(bool)

print('\nAfter:')
print(df.dtypes)
print(f'Memory: {df.memory_usage(deep=True).sum()/1e6:.2f} MB')

Automating Type Optimisation

Write a reusable optimise_dtypes(df) function that automatically applies all optimisations: downcast integers, downcast floats, convert low-cardinality objects to Categorical, and convert 0/1 integers to bool. Run this function at data load time to minimise memory from the start. This ensures every team member who loads the same dataset gets the optimised version without remembering to apply each step manually.

import pandas as pd
import numpy as np

def optimise_dtypes(df, cat_threshold=0.5):
    '''Reduce DataFrame memory by choosing smaller dtypes.'''
    for col in df.columns:
        col_type = df[col].dtype
        if col_type == 'int64':
            df[col] = pd.to_numeric(df[col], downcast='integer')
        elif col_type == 'float64':
            df[col] = pd.to_numeric(df[col], downcast='float')
        elif col_type == 'object':
            cardinality = df[col].nunique() / len(df)
            if cardinality < cat_threshold:
                df[col] = df[col].astype('category')
    return df

# Test
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randint(0,100,500000),
                   'b': np.random.randn(500000),
                   'c': np.random.choice(['X','Y','Z'],500000)})

before = df.memory_usage(deep=True).sum()
df = optimise_dtypes(df)
after = df.memory_usage(deep=True).sum()
print(f'Before: {before/1e6:.2f} MB → After: {after/1e6:.2f} MB ({before/after:.1f}x reduction)')

Unsigned Integer Types for Non-Negative Data

When a column only contains non-negative integers (like IDs, counts, or quantities), use unsigned integer types: uint8 (0–255), uint16 (0–65535), uint32 (0–4 billion), or uint64. Unsigned types have the same byte size as their signed counterparts but can store values up to twice as large in the positive range. For example, a product ID that ranges from 0 to 60,000 fits in uint16 (2 bytes) rather than int32 (4 bytes). Use astype('uint16') after verifying the column has no negative values.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'product_id': np.random.randint(0, 50000, 1000000).astype('int64'),
    'quantity': np.random.randint(0, 255, 1000000).astype('int64')
})

print('Before (int64):', df.memory_usage(deep=True).sum() / 1e6, 'MB')

# product_id fits in uint16 (0-65535)
df['product_id'] = df['product_id'].astype('uint16')
# quantity fits in uint8 (0-255)
df['quantity'] = df['quantity'].astype('uint8')

print('After (uint16+uint8):', df.memory_usage(deep=True).sum() / 1e6, 'MB')
print('\nDtypes:', df.dtypes.to_dict())

Checking Memory After Every Optimisation Step

Apply optimisations one at a time and check memory after each step. This reveals exactly how much each technique contributes. A typical large DataFrame might go from 2 GB (all default dtypes) to 600 MB (integer downcasting), then to 300 MB (float downcasting), then to 200 MB (Categorical for string columns). Documenting this breakdown helps justify the extra complexity to team members who see unfamiliar dtypes in the codebase.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'user_id': np.random.randint(0, 99999, 1000000).astype('int64'),
    'age': np.random.randint(18, 80, 1000000).astype('int64'),
    'revenue': np.random.uniform(0, 5000, 1000000).astype('float64'),
    'country': np.random.choice(['US','UK','DE','FR','JP'], 1000000),
    'tier': np.random.choice(['free','basic','pro','enterprise'], 1000000)
})

def mem_mb(df):
    return df.memory_usage(deep=True).sum() / 1e6

print(f'Start: {mem_mb(df):.1f} MB')

for c in ['user_id','age']:
    df[c] = pd.to_numeric(df[c], downcast='integer')
print(f'After int downcast: {mem_mb(df):.1f} MB')

df['revenue'] = pd.to_numeric(df['revenue'], downcast='float')
print(f'After float downcast: {mem_mb(df):.1f} MB')

for c in ['country','tier']:
    df[c] = df[c].astype('category')
print(f'After Categorical: {mem_mb(df):.1f} MB')

Saving and Reloading Optimised Types

Optimised dtypes are lost if you save to CSV (which converts everything back to text). To preserve dtypes, save to Parquet format with df.to_parquet('file.parquet') — Parquet preserves int32, float32, and Categorical types. When reloaded with pd.read_parquet, the DataFrame has the same memory-efficient types without re-running the optimisation function. Parquet also loads significantly faster than CSV for large files.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'age': np.random.randint(18, 80, 100000).astype('int8'),   # already optimised
    'score': np.random.randn(100000).astype('float32'),
    'tier': pd.Categorical(np.random.choice(['free','pro'], 100000))
})
print('Dtypes:', df.dtypes.to_dict())

# Save to Parquet (preserves dtypes)
df.to_parquet('/tmp/optimised.parquet', index=False)

# Reload — dtypes preserved!
df_loaded = pd.read_parquet('/tmp/optimised.parquet')
print('\nLoaded dtypes:', df_loaded.dtypes.to_dict())
print(f'Memory: {df_loaded.memory_usage(deep=True).sum()/1e6:.2f} MB')

Quick Check

Test your understanding of memory-efficient data types from this lesson.

Lesson Recap

In this lesson you learned: pd.to_numeric(downcast='integer') reduces integer columns from 8 bytes to 1-4 bytes, pd.to_numeric(downcast='float') halves float memory, Categorical dtype reduces low-cardinality string columns by up to 50x while also speeding up groupby, and Parquet format preserves optimised dtypes across file save/load cycles. Next up we explore chunked reading — processing files that exceed available RAM.

Frequently asked questions

Is the “Efficient Data Types for Memory Reduction” lesson free?

Yes — the full text of “Efficient Data Types for Memory Reduction” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.

What will I learn in “Efficient Data Types for Memory Reduction”?

Downcast numeric columns to int32/float32 and convert string columns to Categorical to cut DataFrame memory by up to 70%. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Pandas & NumPy Academy?

No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Efficient Data Types for Memory Reduction” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Pandas & NumPy Academy lesson?

Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Profiling with timeit and memory_profiler
  2. Avoiding iterrows and Python Loops
  3. Efficient Data Types for Memory Reduction
  4. Chunked Reading for Large Files
← Back to Pandas & NumPy Academy