Profiling with timeit and memory_profiler
Measure execution time with %timeit and peak memory with memory_profiler to identify the slowest parts of your pipeline.
Profiling with timeit and memory_profiler is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Profile Before Optimising?
A common mistake is to optimise code intuitively — rewriting a function that looks slow without measuring whether it is actually the bottleneck. In data pipelines, 90% of the execution time often comes from 10% of the code. Profiling identifies exactly where time and memory are consumed so you can focus optimisation effort where it matters. The golden rule: measure first, optimise second. Tools like timeit measure time and memory_profiler measures RAM.
import pandas as pd
import numpy as np
import timeit
# Naive approach vs vectorised approach — which is faster?
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(100000), 'b': np.random.randn(100000)})
t1 = timeit.timeit(lambda: df['a'] + df['b'], number=100)
t2 = timeit.timeit(lambda: [a + b for a, b in zip(df['a'], df['b'])], number=10)
print(f'Vectorised (100 runs): {t1:.3f}s')
print(f'List comprehension (10 runs): {t2:.3f}s')
print(f'Per-run speedup: ~{(t2/10)/(t1/100):.0f}x')timeit: Python's Built-In Timer
The timeit module measures execution time by running a statement many times and returning the total elapsed time. timeit.timeit(stmt, number=n) runs stmt n times and returns total seconds. Divide by number to get per-run time. For accurate measurements, choose number so the total run takes at least 0.1 seconds — a single run of a fast operation is not representative due to system scheduling noise.
import timeit
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(10000)})
# Compare three ways to square a column
time_a = timeit.timeit(lambda: df['x'] ** 2, number=1000)
time_b = timeit.timeit(lambda: df['x'].apply(lambda v: v**2), number=100)
time_c = timeit.timeit(lambda: np.square(df['x']), number=1000)
print(f'Operator **2 per run: {time_a/1000*1000:.3f} ms')
print(f'.apply(v**2) per run: {time_b/100*1000:.3f} ms')
print(f'np.square() per run: {time_c/1000*1000:.3f} ms')%timeit Magic in Jupyter
In Jupyter notebooks, the %timeit magic automatically selects the number of repetitions and runs to give a statistically stable estimate, then reports the best and mean times with standard deviation. It is the most convenient way to quickly benchmark a single line. Use %%timeit (double percent) to time a whole cell. The output format is: X ns/μs/ms ± Y ns per loop (mean ± std. dev. of 7 runs, N loops each).
# In Jupyter notebook:
# %timeit df['x'] ** 2
# Output: 47.3 us +- 1.2 us per loop (mean +- std. dev. of 7 runs, 10000 loops each)
# Multi-line cell timing:
# %%timeit
# result = df['x'] ** 2
# total = result.sum()
# In a regular Python script, use timeit.timeit() instead:
import timeit
import pandas as pd, numpy as np
df = pd.DataFrame({'x': np.random.randn(10000)})
result = timeit.repeat(lambda: df['x']**2, number=1000, repeat=7)
print(f'Best run: {min(result)/1000*1000:.3f} ms per call')cProfile for Function-Level Profiling
While timeit measures a single expression, cProfile profiles the entire call stack — it shows how much time is spent in every function called during execution. This is essential for identifying the bottleneck in a complex pipeline. Run python -m cProfile -s cumulative script.py from the command line, or use cProfile.run('function()') in a script. In Jupyter, use %prun function().
import cProfile
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(50000), 'b': np.random.randn(50000)})
def slow_pipeline(df):
result = []
for _, row in df.iterrows():
result.append(row['a'] * row['b'])
return pd.Series(result)
# Profile the function
print('Profile output (top 5 functions by cumulative time):')
cProfile.run('slow_pipeline(df.head(1000))', sort='cumulative')Measuring Memory Usage
Use df.memory_usage(deep=True) to see how much RAM each column consumes. The deep=True argument measures the actual memory used by object dtype columns (strings), which otherwise reports only the pointer size. Use .sum() to get the total DataFrame memory in bytes. Divide by 1e6 for MB or 1e9 for GB. This is the starting point for memory optimisation — you cannot reduce what you have not measured.
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({
'id': range(100000),
'name': ['user_' + str(i) for i in range(100000)],
'score': np.random.randn(100000),
'category': np.random.choice(['A', 'B', 'C', 'D'], 100000)
})
mem = df.memory_usage(deep=True)
print('Memory per column (bytes):')
print(mem)
print(f'\nTotal: {mem.sum() / 1e6:.2f} MB')memory_profiler for Line-by-Line Memory
The memory_profiler package (pip install memory-profiler) measures RAM usage line by line in a function. Decorate a function with @profile and run it with python -m memory_profiler script.py. In Jupyter, use the %memit magic (from the ipython extension) to measure peak memory of a single expression. This identifies which data structures cause sudden memory spikes — often concatenating in a loop instead of collecting and concatenating once.
# Install: pip install memory-profiler
# Usage as a decorator:
# from memory_profiler import profile
# @profile
# def process_data():
# import pandas as pd, numpy as np
# df = pd.DataFrame({'x': np.random.randn(1000000)})
# result = df['x'].rolling(100).mean() # <-- sees if this spikes RAM
# return result
# Run: python -m memory_profiler script.py
# In Jupyter (after %load_ext memory_profiler):
# %memit pd.DataFrame({'x': np.random.randn(1000000)}).rolling(100).mean()
print('memory_profiler decorates functions for line-by-line RAM measurement.')
print('Typical output: Line # Mem usage Increment Line Contents')sys.getsizeof vs memory_usage for Objects
sys.getsizeof(obj) returns the memory size of a Python object, but for pandas DataFrames it only reports the container overhead (a few hundred bytes), not the actual data. Always use df.memory_usage(deep=True).sum() for DataFrames. For individual Python objects like lists and dicts, sys.getsizeof is accurate but does not recurse into nested objects — use the pympler library (asizeof) for deep object sizes.
import sys
import pandas as pd
import numpy as np
df = pd.DataFrame({'x': np.random.randn(100000)})
# sys.getsizeof reports only the container metadata — misleading!
print('sys.getsizeof(df):', sys.getsizeof(df), 'bytes (misleading!)')
# memory_usage gives the true memory
true_mem = df.memory_usage(deep=True).sum()
print('df.memory_usage(deep=True).sum():', true_mem, 'bytes')
print(f'True memory: {true_mem/1e6:.2f} MB')Timing Context Manager Pattern
For longer pipeline steps where timeit is not suitable (e.g. database queries or file I/O), use a simple context manager with time.perf_counter(). Wrap each pipeline step in a timer context to log its duration. This is more practical than timeit for production code because it logs real wall-clock time including I/O wait, which timeit also measures but is harder to integrate with logging frameworks.
import time
import contextlib
import pandas as pd
import numpy as np
@contextlib.contextmanager
def timer(name):
start = time.perf_counter()
yield
elapsed = time.perf_counter() - start
print(f'[{name}] {elapsed*1000:.1f} ms')
np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(500000), 'y': np.random.randn(500000)})
with timer('rolling mean'):
result1 = df['x'].rolling(30).mean()
with timer('groupby mean'):
df['group'] = df['x'].round(0)
result2 = df.groupby('group')['y'].mean()Profiling Pandas Operations Systematically
A systematic profiling workflow: 1) Start with a representative sample of your data (10,000 rows is enough for timing comparisons). 2) Identify the slowest steps using a timer context or cProfile. 3) Rewrite the slowest step using a faster approach (vectorisation, Categorical dtype, C extension). 4) Verify correctness with assert statements. 5) Re-time to confirm the speedup. 6) Repeat for the next bottleneck until the pipeline meets its SLA.
import pandas as pd
import numpy as np
import timeit
np.random.seed(0)
df = pd.DataFrame({
'value': np.random.randn(100000),
'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 100000)
})
# Benchmark two groupby implementations
t1 = timeit.timeit(lambda: df.groupby('category')['value'].mean(), number=100)
df2 = df.copy()
df2['category'] = df2['category'].astype('category')
t2 = timeit.timeit(lambda: df2.groupby('category')['value'].mean(), number=100)
print(f'Object dtype groupby: {t1/100*1000:.2f} ms per call')
print(f'Categorical dtype: {t2/100*1000:.2f} ms per call')
print(f'Speedup: {t1/t2:.1f}x')When to Profile: Practical Thresholds
Not every script needs profiling. Invest in profiling when: a pipeline takes more than 30 seconds to run and runs frequently (daily or more), a function processes more than 100,000 rows, or memory usage exceeds 50% of available RAM. For scripts that run once or in seconds, readability matters more than microsecond optimisation. The cost of maintaining heavily optimised, obscure code often exceeds the time saved by the optimisation.
Snapshot: Profiling Tools Summary
A quick reference:
- timeit.timeit — measure time of a single statement (script)
- %timeit / %%timeit — measure time in Jupyter (auto-repeats)
- %prun / cProfile — function-level call stack profiling
- df.memory_usage(deep=True) — per-column RAM measurement
- memory_profiler @profile — line-by-line RAM measurement
- time.perf_counter() — manual timer for pipeline steps
Quick Check
Test your understanding of profiling tools from this lesson.
Lesson Recap
In this lesson you learned: timeit.timeit and %timeit measure execution time of specific operations, df.memory_usage(deep=True) measures per-column RAM accurately, and cProfile / memory_profiler provide deeper profiling for complex pipelines. Always profile before optimising, and verify correctness after each change. Next up we explore how to avoid slow iterrows loops and replace them with vectorised operations.
Frequently asked questions
Is the “Profiling with timeit and memory_profiler” lesson free?
Yes — the full text of “Profiling with timeit and memory_profiler” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Profiling with timeit and memory_profiler”?
Measure execution time with %timeit and peak memory with memory_profiler to identify the slowest parts of your pipeline. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Profiling with timeit and memory_profiler” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Profiling with timeit and memory_profiler
- Avoiding iterrows and Python Loops
- Efficient Data Types for Memory Reduction
- Chunked Reading for Large Files