Profiling mit timeit und memory_profiler
Messen Sie die Ausführungszeit mit %timeit und den Spitzenverbrauch des Arbeitsspeichers mit memory_profiler, um die langsamsten Teile Ihrer Pipeline zu ermitteln.
Profiling mit timeit und memory_profiler ist eine kostenlose Pandas & NumPy Academy-Lektion auf CoddyKit. Dies ist Lektion 1 von 4. Du kannst die komplette Lektion unten kostenlos lesen – dann übst du sie direkt im Browser mit einem integrierten Code-Editor und einem KI-Tutor rund um die Uhr. Sie ist Teil des Pandas & NumPy Academy-Lernpfads, und dein Fortschritt wird über Web und CoddyKit-App synchronisiert. Der Pandas & NumPy Academy-Kurs umfasst insgesamt 4 Lektionen.
Teile dieser Lektion wurden noch nicht übersetzt und werden auf Englisch angezeigt.
Why Profile Before Optimising?
A common mistake is to optimise code intuitively — rewriting a function that looks slow without measuring whether it is actually the bottleneck. In data pipelines, 90% of the execution time often comes from 10% of the code. Profiling identifies exactly where time and memory are consumed so you can focus optimisation effort where it matters. The golden rule: measure first, optimise second. Tools like timeit measure time and memory_profiler measures RAM.
import pandas as pd
import numpy as np
import timeit
# Naive approach vs vectorised approach — which is faster?
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(100000), 'b': np.random.randn(100000)})
t1 = timeit.timeit(lambda: df['a'] + df['b'], number=100)
t2 = timeit.timeit(lambda: [a + b for a, b in zip(df['a'], df['b'])], number=10)
print(f'Vectorised (100 runs): {t1:.3f}s')
print(f'List comprehension (10 runs): {t2:.3f}s')
print(f'Per-run speedup: ~{(t2/10)/(t1/100):.0f}x')timeit: Python's Built-In Timer
The timeit module measures execution time by running a statement many times and returning the total elapsed time. timeit.timeit(stmt, number=n) runs stmt n times and returns total seconds. Divide by number to get per-run time. For accurate measurements, choose number so the total run takes at least 0.1 seconds — a single run of a fast operation is not representative due to system scheduling noise.
import timeit
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(10000)})
# Compare three ways to square a column
time_a = timeit.timeit(lambda: df['x'] ** 2, number=1000)
time_b = timeit.timeit(lambda: df['x'].apply(lambda v: v**2), number=100)
time_c = timeit.timeit(lambda: np.square(df['x']), number=1000)
print(f'Operator **2 per run: {time_a/1000*1000:.3f} ms')
print(f'.apply(v**2) per run: {time_b/100*1000:.3f} ms')
print(f'np.square() per run: {time_c/1000*1000:.3f} ms')%timeit Magic in Jupyter
In Jupyter notebooks, the %timeit magic automatically selects the number of repetitions and runs to give a statistically stable estimate, then reports the best and mean times with standard deviation. It is the most convenient way to quickly benchmark a single line. Use %%timeit (double percent) to time a whole cell. The output format is: X ns/μs/ms ± Y ns per loop (mean ± std. dev. of 7 runs, N loops each).
# In Jupyter notebook:
# %timeit df['x'] ** 2
# Output: 47.3 us +- 1.2 us per loop (mean +- std. dev. of 7 runs, 10000 loops each)
# Multi-line cell timing:
# %%timeit
# result = df['x'] ** 2
# total = result.sum()
# In a regular Python script, use timeit.timeit() instead:
import timeit
import pandas as pd, numpy as np
df = pd.DataFrame({'x': np.random.randn(10000)})
result = timeit.repeat(lambda: df['x']**2, number=1000, repeat=7)
print(f'Best run: {min(result)/1000*1000:.3f} ms per call')cProfile for Function-Level Profiling
While timeit measures a single expression, cProfile profiles the entire call stack — it shows how much time is spent in every function called during execution. This is essential for identifying the bottleneck in a complex pipeline. Run python -m cProfile -s cumulative script.py from the command line, or use cProfile.run('function()') in a script. In Jupyter, use %prun function().
import cProfile
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(50000), 'b': np.random.randn(50000)})
def slow_pipeline(df):
result = []
for _, row in df.iterrows():
result.append(row['a'] * row['b'])
return pd.Series(result)
# Profile the function
print('Profile output (top 5 functions by cumulative time):')
cProfile.run('slow_pipeline(df.head(1000))', sort='cumulative')Measuring Memory Usage
Use df.memory_usage(deep=True) to see how much RAM each column consumes. The deep=True argument measures the actual memory used by object dtype columns (strings), which otherwise reports only the pointer size. Use .sum() to get the total DataFrame memory in bytes. Divide by 1e6 for MB or 1e9 for GB. This is the starting point for memory optimisation — you cannot reduce what you have not measured.
import pandas as pd
import numpy as np
np.random.seed(0)
df = pd.DataFrame({
'id': range(100000),
'name': ['user_' + str(i) for i in range(100000)],
'score': np.random.randn(100000),
'category': np.random.choice(['A', 'B', 'C', 'D'], 100000)
})
mem = df.memory_usage(deep=True)
print('Memory per column (bytes):')
print(mem)
print(f'\nTotal: {mem.sum() / 1e6:.2f} MB')memory_profiler for Line-by-Line Memory
The memory_profiler package (pip install memory-profiler) measures RAM usage line by line in a function. Decorate a function with @profile and run it with python -m memory_profiler script.py. In Jupyter, use the %memit magic (from the ipython extension) to measure peak memory of a single expression. This identifies which data structures cause sudden memory spikes — often concatenating in a loop instead of collecting and concatenating once.
# Install: pip install memory-profiler
# Usage as a decorator:
# from memory_profiler import profile
# @profile
# def process_data():
# import pandas as pd, numpy as np
# df = pd.DataFrame({'x': np.random.randn(1000000)})
# result = df['x'].rolling(100).mean() # <-- sees if this spikes RAM
# return result
# Run: python -m memory_profiler script.py
# In Jupyter (after %load_ext memory_profiler):
# %memit pd.DataFrame({'x': np.random.randn(1000000)}).rolling(100).mean()
print('memory_profiler decorates functions for line-by-line RAM measurement.')
print('Typical output: Line # Mem usage Increment Line Contents')sys.getsizeof vs memory_usage for Objects
sys.getsizeof(obj) returns the memory size of a Python object, but for pandas DataFrames it only reports the container overhead (a few hundred bytes), not the actual data. Always use df.memory_usage(deep=True).sum() for DataFrames. For individual Python objects like lists and dicts, sys.getsizeof is accurate but does not recurse into nested objects — use the pympler library (asizeof) for deep object sizes.
import sys
import pandas as pd
import numpy as np
df = pd.DataFrame({'x': np.random.randn(100000)})
# sys.getsizeof reports only the container metadata — misleading!
print('sys.getsizeof(df):', sys.getsizeof(df), 'bytes (misleading!)')
# memory_usage gives the true memory
true_mem = df.memory_usage(deep=True).sum()
print('df.memory_usage(deep=True).sum():', true_mem, 'bytes')
print(f'True memory: {true_mem/1e6:.2f} MB')Timing Context Manager Pattern
For longer pipeline steps where timeit is not suitable (e.g. database queries or file I/O), use a simple context manager with time.perf_counter(). Wrap each pipeline step in a timer context to log its duration. This is more practical than timeit for production code because it logs real wall-clock time including I/O wait, which timeit also measures but is harder to integrate with logging frameworks.
import time
import contextlib
import pandas as pd
import numpy as np
@contextlib.contextmanager
def timer(name):
start = time.perf_counter()
yield
elapsed = time.perf_counter() - start
print(f'[{name}] {elapsed*1000:.1f} ms')
np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(500000), 'y': np.random.randn(500000)})
with timer('rolling mean'):
result1 = df['x'].rolling(30).mean()
with timer('groupby mean'):
df['group'] = df['x'].round(0)
result2 = df.groupby('group')['y'].mean()Profiling Pandas Operations Systematically
A systematic profiling workflow: 1) Start with a representative sample of your data (10,000 rows is enough for timing comparisons). 2) Identify the slowest steps using a timer context or cProfile. 3) Rewrite the slowest step using a faster approach (vectorisation, Categorical dtype, C extension). 4) Verify correctness with assert statements. 5) Re-time to confirm the speedup. 6) Repeat for the next bottleneck until the pipeline meets its SLA.
import pandas as pd
import numpy as np
import timeit
np.random.seed(0)
df = pd.DataFrame({
'value': np.random.randn(100000),
'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 100000)
})
# Benchmark two groupby implementations
t1 = timeit.timeit(lambda: df.groupby('category')['value'].mean(), number=100)
df2 = df.copy()
df2['category'] = df2['category'].astype('category')
t2 = timeit.timeit(lambda: df2.groupby('category')['value'].mean(), number=100)
print(f'Object dtype groupby: {t1/100*1000:.2f} ms per call')
print(f'Categorical dtype: {t2/100*1000:.2f} ms per call')
print(f'Speedup: {t1/t2:.1f}x')When to Profile: Practical Thresholds
Not every script needs profiling. Invest in profiling when: a pipeline takes more than 30 seconds to run and runs frequently (daily or more), a function processes more than 100,000 rows, or memory usage exceeds 50% of available RAM. For scripts that run once or in seconds, readability matters more than microsecond optimisation. The cost of maintaining heavily optimised, obscure code often exceeds the time saved by the optimisation.
Snapshot: Profiling Tools Summary
A quick reference:
- timeit.timeit — measure time of a single statement (script)
- %timeit / %%timeit — measure time in Jupyter (auto-repeats)
- %prun / cProfile — function-level call stack profiling
- df.memory_usage(deep=True) — per-column RAM measurement
- memory_profiler @profile — line-by-line RAM measurement
- time.perf_counter() — manual timer for pipeline steps
Quick Check
Test your understanding of profiling tools from this lesson.
Lesson Recap
In this lesson you learned: timeit.timeit and %timeit measure execution time of specific operations, df.memory_usage(deep=True) measures per-column RAM accurately, and cProfile / memory_profiler provide deeper profiling for complex pipelines. Always profile before optimising, and verify correctness after each change. Next up we explore how to avoid slow iterrows loops and replace them with vectorised operations.
Häufig gestellte Fragen
Ist die Lektion „Profiling mit timeit und memory_profiler“ kostenlos?
Ja — der vollständige Text von „Profiling mit timeit und memory_profiler“ ist hier im Web kostenlos zu lesen. Um sie interaktiv zu üben (integrierter Code-Editor und 24/7 KI-Tutor) und den Rest des Pandas & NumPy Academy-Kurses freizuschalten, upgrade auf CoddyKit PRO. Der Pandas & NumPy Academy-Kurs umfasst insgesamt 4 Lektionen.
Was lerne ich in „Profiling mit timeit und memory_profiler“?
Messen Sie die Ausführungszeit mit %timeit und den Spitzenverbrauch des Arbeitsspeichers mit memory_profiler, um die langsamsten Teile Ihrer Pipeline zu ermitteln. Du übst Pandas & NumPy Academy mit praktischem Code, den du direkt im Browser ausführst, und ein 24/7 KI-Tutor beantwortet deine Fragen während du die Lektion bearbeitest.
Brauche ich Erfahrung, um Pandas & NumPy Academy zu starten?
Keine Vorkenntnisse erforderlich. Pandas & NumPy Academy auf CoddyKit ist für Anfänger bis fortgeschrittene Lernende strukturiert, sodass du hier starten oder von Anfang an beginnen und in deinem eigenen Tempo voranschreiten kannst. Dies ist Lektion 1 von 4.
Wie lange dauert die Lektion „Profiling mit timeit und memory_profiler“?
Die meisten CoddyKit-Lektionen dauern etwa 5–10 Minuten. Jede ist kompakt und interaktiv, sodass du stetig Fortschritte machst und genau dort weitermachst, wo du aufgehört hast – im Web und in der App.
Kann ich in dieser Pandas & NumPy Academy-Lektion Code schreiben und ausführen?
Ja. Jede Pandas & NumPy Academy-Lektion enthält einen integrierten Code-Editor, sodass du echten Code direkt in deinem Browser schreibst und ausführst und sofort KI-Feedback erhältst — ohne lokale Einrichtung erforderlich.
Alle Lektionen in diesem Kurs
- Profiling mit timeit und memory_profiler
- iterrows und Python-Schleifen vermeiden
- Effiziente Datentypen zur Speicherreduzierung
- Chunkweises Lesen großer Dateien