Profilage avec timeit et memory_profiler
Mesurez le temps d’exécution avec %timeit et la mémoire maximale avec memory_profiler pour repérer les parties les plus lentes de votre pipeline.
Profilage avec timeit et memory_profiler est une leçon Pandas & NumPy Academy gratuite sur CoddyKit. Ceci est la leçon 1 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage Pandas & NumPy Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours Pandas & NumPy Academy comprend 4 leçons au total.
Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.
Why Profile Before Optimising?
A common mistake is to optimise code intuitively — rewriting a function that looks slow without measuring whether it is actually the bottleneck. In data pipelines, 90% of the execution time often comes from 10% of the code. Profiling identifies exactly where time and memory are consumed so you can focus optimisation effort where it matters. The golden rule: measure first, optimise second. Tools like timeit measure time and memory_profiler measures RAM.
import pandas as pd
import numpy as np
import timeit
# Naive approach vs vectorised approach — which is faster?
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(100000), 'b': np.random.randn(100000)})
t1 = timeit.timeit(lambda: df['a'] + df['b'], number=100)
t2 = timeit.timeit(lambda: [a + b for a, b in zip(df['a'], df['b'])], number=10)
print(f'Vectorised (100 runs): {t1:.3f}s')
print(f'List comprehension (10 runs): {t2:.3f}s')
print(f'Per-run speedup: ~{(t2/10)/(t1/100):.0f}x')timeit: Python's Built-In Timer
The timeit module measures execution time by running a statement many times and returning the total elapsed time. timeit.timeit(stmt, number=n) runs stmt n times and returns total seconds. Divide by number to get per-run time. For accurate measurements, choose number so the total run takes at least 0.1 seconds — a single run of a fast operation is not representative due to system scheduling noise.
import timeit
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(10000)})
# Compare three ways to square a column
time_a = timeit.timeit(lambda: df['x'] ** 2, number=1000)
time_b = timeit.timeit(lambda: df['x'].apply(lambda v: v**2), number=100)
time_c = timeit.timeit(lambda: np.square(df['x']), number=1000)
print(f'Operator **2 per run: {time_a/1000*1000:.3f} ms')
print(f'.apply(v**2) per run: {time_b/100*1000:.3f} ms')
print(f'np.square() per run: {time_c/1000*1000:.3f} ms')%timeit Magic in Jupyter
In Jupyter notebooks, the %timeit magic automatically selects the number of repetitions and runs to give a statistically stable estimate, then reports the best and mean times with standard deviation. It is the most convenient way to quickly benchmark a single line. Use %%timeit (double percent) to time a whole cell. The output format is: X ns/μs/ms ± Y ns per loop (mean ± std. dev. of 7 runs, N loops each).
# In Jupyter notebook:
# %timeit df['x'] ** 2
# Output: 47.3 us +- 1.2 us per loop (mean +- std. dev. of 7 runs, 10000 loops each)
# Multi-line cell timing:
# %%timeit
# result = df['x'] ** 2
# total = result.sum()
# In a regular Python script, use timeit.timeit() instead:
import timeit
import pandas as pd, numpy as np
df = pd.DataFrame({'x': np.random.randn(10000)})
result = timeit.repeat(lambda: df['x']**2, number=1000, repeat=7)
print(f'Best run: {min(result)/1000*1000:.3f} ms per call')cProfile for Function-Level Profiling
While timeit measures a single expression, cProfile profiles the entire call stack — it shows how much time is spent in every function called during execution. This is essential for identifying the bottleneck in a complex pipeline. Run python -m cProfile -s cumulative script.py from the command line, or use cProfile.run('function()') in a script. In Jupyter, use %prun function().
import cProfile
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(50000), 'b': np.random.randn(50000)})
def slow_pipeline(df):
result = []
for _, row in df.iterrows():
result.append(row['a'] * row['b'])
return pd.Series(result)
# Profile the function
print('Profile output (top 5 functions by cumulative time):')
cProfile.run('slow_pipeline(df.head(1000))', sort='cumulative')Measuring Memory Usage
Use df.memory_usage(deep=True) to see how much RAM each column consumes. The deep=True argument measures the actual memory used by object dtype columns (strings), which otherwise reports only the pointer size. Use .sum() to get the total DataFrame memory in bytes. Divide by 1e6 for MB or 1e9 for GB. This is the starting point for memory optimisation — you cannot reduce what you have not measured.
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({
'id': range(100000),
'name': ['user_' + str(i) for i in range(100000)],
'score': np.random.randn(100000),
'category': np.random.choice(['A', 'B', 'C', 'D'], 100000)
})
mem = df.memory_usage(deep=True)
print('Memory per column (bytes):')
print(mem)
print(f'\nTotal: {mem.sum() / 1e6:.2f} MB')memory_profiler for Line-by-Line Memory
The memory_profiler package (pip install memory-profiler) measures RAM usage line by line in a function. Decorate a function with @profile and run it with python -m memory_profiler script.py. In Jupyter, use the %memit magic (from the ipython extension) to measure peak memory of a single expression. This identifies which data structures cause sudden memory spikes — often concatenating in a loop instead of collecting and concatenating once.
# Install: pip install memory-profiler
# Usage as a decorator:
# from memory_profiler import profile
# @profile
# def process_data():
# import pandas as pd, numpy as np
# df = pd.DataFrame({'x': np.random.randn(1000000)})
# result = df['x'].rolling(100).mean() # <-- sees if this spikes RAM
# return result
# Run: python -m memory_profiler script.py
# In Jupyter (after %load_ext memory_profiler):
# %memit pd.DataFrame({'x': np.random.randn(1000000)}).rolling(100).mean()
print('memory_profiler decorates functions for line-by-line RAM measurement.')
print('Typical output: Line # Mem usage Increment Line Contents')sys.getsizeof vs memory_usage for Objects
sys.getsizeof(obj) returns the memory size of a Python object, but for pandas DataFrames it only reports the container overhead (a few hundred bytes), not the actual data. Always use df.memory_usage(deep=True).sum() for DataFrames. For individual Python objects like lists and dicts, sys.getsizeof is accurate but does not recurse into nested objects — use the pympler library (asizeof) for deep object sizes.
import sys
import pandas as pd
import numpy as np
df = pd.DataFrame({'x': np.random.randn(100000)})
# sys.getsizeof reports only the container metadata — misleading!
print('sys.getsizeof(df):', sys.getsizeof(df), 'bytes (misleading!)')
# memory_usage gives the true memory
true_mem = df.memory_usage(deep=True).sum()
print('df.memory_usage(deep=True).sum():', true_mem, 'bytes')
print(f'True memory: {true_mem/1e6:.2f} MB')Timing Context Manager Pattern
For longer pipeline steps where timeit is not suitable (e.g. database queries or file I/O), use a simple context manager with time.perf_counter(). Wrap each pipeline step in a timer context to log its duration. This is more practical than timeit for production code because it logs real wall-clock time including I/O wait, which timeit also measures but is harder to integrate with logging frameworks.
import time
import contextlib
import pandas as pd
import numpy as np
@contextlib.contextmanager
def timer(name):
start = time.perf_counter()
yield
elapsed = time.perf_counter() - start
print(f'[{name}] {elapsed*1000:.1f} ms')
np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(500000), 'y': np.random.randn(500000)})
with timer('rolling mean'):
result1 = df['x'].rolling(30).mean()
with timer('groupby mean'):
df['group'] = df['x'].round(0)
result2 = df.groupby('group')['y'].mean()Profiling Pandas Operations Systematically
A systematic profiling workflow: 1) Start with a representative sample of your data (10,000 rows is enough for timing comparisons). 2) Identify the slowest steps using a timer context or cProfile. 3) Rewrite the slowest step using a faster approach (vectorisation, Categorical dtype, C extension). 4) Verify correctness with assert statements. 5) Re-time to confirm the speedup. 6) Repeat for the next bottleneck until the pipeline meets its SLA.
import pandas as pd
import numpy as np
import timeit
np.random.seed(0)
df = pd.DataFrame({
'value': np.random.randn(100000),
'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 100000)
})
# Benchmark two groupby implementations
t1 = timeit.timeit(lambda: df.groupby('category')['value'].mean(), number=100)
df2 = df.copy()
df2['category'] = df2['category'].astype('category')
t2 = timeit.timeit(lambda: df2.groupby('category')['value'].mean(), number=100)
print(f'Object dtype groupby: {t1/100*1000:.2f} ms per call')
print(f'Categorical dtype: {t2/100*1000:.2f} ms per call')
print(f'Speedup: {t1/t2:.1f}x')When to Profile: Practical Thresholds
Not every script needs profiling. Invest in profiling when: a pipeline takes more than 30 seconds to run and runs frequently (daily or more), a function processes more than 100,000 rows, or memory usage exceeds 50% of available RAM. For scripts that run once or in seconds, readability matters more than microsecond optimisation. The cost of maintaining heavily optimised, obscure code often exceeds the time saved by the optimisation.
Snapshot: Profiling Tools Summary
A quick reference:
- timeit.timeit — measure time of a single statement (script)
- %timeit / %%timeit — measure time in Jupyter (auto-repeats)
- %prun / cProfile — function-level call stack profiling
- df.memory_usage(deep=True) — per-column RAM measurement
- memory_profiler @profile — line-by-line RAM measurement
- time.perf_counter() — manual timer for pipeline steps
Quick Check
Test your understanding of profiling tools from this lesson.
Lesson Recap
In this lesson you learned: timeit.timeit and %timeit measure execution time of specific operations, df.memory_usage(deep=True) measures per-column RAM accurately, and cProfile / memory_profiler provide deeper profiling for complex pipelines. Always profile before optimising, and verify correctness after each change. Next up we explore how to avoid slow iterrows loops and replace them with vectorised operations.
Questions Fréquemment Posées
La leçon « Profilage avec timeit et memory_profiler » est-elle gratuite ?
Oui — le texte complet de « Profilage avec timeit et memory_profiler » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours Pandas & NumPy Academy, passe à CoddyKit PRO. Le cours Pandas & NumPy Academy comprend 4 leçons au total.
Qu'est-ce que j'apprendrai dans « Profilage avec timeit et memory_profiler » ?
Mesurez le temps d’exécution avec %timeit et la mémoire maximale avec memory_profiler pour repérer les parties les plus lentes de votre pipeline. Tu pratiques Pandas & NumPy Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.
Dois-je avoir de l'expérience pour commencer Pandas & NumPy Academy ?
Aucune expérience préalable n'est requise. Pandas & NumPy Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 1 sur 4.
Combien de temps prend la leçon « Profilage avec timeit et memory_profiler » ?
La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.
Peux-tu écrire et exécuter du code dans cette leçon Pandas & NumPy Academy ?
Oui. Chaque leçon Pandas & NumPy Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.
Toutes les leçons de ce cours
- Profilage avec timeit et memory_profiler
- Éviter iterrows et les boucles Python
- Types de données efficaces pour réduire la mémoire
- Lecture par blocs de fichiers volumineux