0Pricing
Pandas & NumPy Academy · 课时

使用 timeit 和 memory_profiler 进行性能分析

使用 %timeit 测量执行时间，使用 memory_profiler 测量峰值内存，从而找出 pipeline 中最慢的部分。

使用 timeit 和 memory_profiler 进行性能分析 是 CoddyKit 上的免费 Pandas & NumPy Academy 课时。 这是第 1 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 Pandas & NumPy Academy 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 Pandas & NumPy Academy 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Profile Before Optimising?

A common mistake is to optimise code intuitively — rewriting a function that looks slow without measuring whether it is actually the bottleneck. In data pipelines, 90% of the execution time often comes from 10% of the code. Profiling identifies exactly where time and memory are consumed so you can focus optimisation effort where it matters. The golden rule: measure first, optimise second. Tools like timeit measure time and memory_profiler measures RAM.

import pandas as pd
import numpy as np
import timeit

# Naive approach vs vectorised approach — which is faster?
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(100000), 'b': np.random.randn(100000)})

t1 = timeit.timeit(lambda: df['a'] + df['b'], number=100)
t2 = timeit.timeit(lambda: [a + b for a, b in zip(df['a'], df['b'])], number=10)

print(f'Vectorised (100 runs): {t1:.3f}s')
print(f'List comprehension (10 runs): {t2:.3f}s')
print(f'Per-run speedup: ~{(t2/10)/(t1/100):.0f}x')

timeit: Python's Built-In Timer

The timeit module measures execution time by running a statement many times and returning the total elapsed time. timeit.timeit(stmt, number=n) runs stmt n times and returns total seconds. Divide by number to get per-run time. For accurate measurements, choose number so the total run takes at least 0.1 seconds — a single run of a fast operation is not representative due to system scheduling noise.

import timeit
import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(10000)})

# Compare three ways to square a column
time_a = timeit.timeit(lambda: df['x'] ** 2, number=1000)
time_b = timeit.timeit(lambda: df['x'].apply(lambda v: v**2), number=100)
time_c = timeit.timeit(lambda: np.square(df['x']), number=1000)

print(f'Operator **2  per run: {time_a/1000*1000:.3f} ms')
print(f'.apply(v**2) per run: {time_b/100*1000:.3f} ms')
print(f'np.square()  per run: {time_c/1000*1000:.3f} ms')

%timeit Magic in Jupyter

In Jupyter notebooks, the %timeit magic automatically selects the number of repetitions and runs to give a statistically stable estimate, then reports the best and mean times with standard deviation. It is the most convenient way to quickly benchmark a single line. Use %%timeit (double percent) to time a whole cell. The output format is: X ns/μs/ms ± Y ns per loop (mean ± std. dev. of 7 runs, N loops each).

# In Jupyter notebook:
# %timeit df['x'] ** 2
# Output: 47.3 us +- 1.2 us per loop (mean +- std. dev. of 7 runs, 10000 loops each)

# Multi-line cell timing:
# %%timeit
# result = df['x'] ** 2
# total = result.sum()

# In a regular Python script, use timeit.timeit() instead:
import timeit
import pandas as pd, numpy as np
df = pd.DataFrame({'x': np.random.randn(10000)})
result = timeit.repeat(lambda: df['x']**2, number=1000, repeat=7)
print(f'Best run: {min(result)/1000*1000:.3f} ms per call')

cProfile for Function-Level Profiling

While timeit measures a single expression, cProfile profiles the entire call stack — it shows how much time is spent in every function called during execution. This is essential for identifying the bottleneck in a complex pipeline. Run python -m cProfile -s cumulative script.py from the command line, or use cProfile.run('function()') in a script. In Jupyter, use %prun function().

import cProfile
import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(50000), 'b': np.random.randn(50000)})

def slow_pipeline(df):
    result = []
    for _, row in df.iterrows():
        result.append(row['a'] * row['b'])
    return pd.Series(result)

# Profile the function
print('Profile output (top 5 functions by cumulative time):')
cProfile.run('slow_pipeline(df.head(1000))', sort='cumulative')

Measuring Memory Usage

Use df.memory_usage(deep=True) to see how much RAM each column consumes. The deep=True argument measures the actual memory used by object dtype columns (strings), which otherwise reports only the pointer size. Use .sum() to get the total DataFrame memory in bytes. Divide by 1e6 for MB or 1e9 for GB. This is the starting point for memory optimisation — you cannot reduce what you have not measured.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'id': range(100000),
    'name': ['user_' + str(i) for i in range(100000)],
    'score': np.random.randn(100000),
    'category': np.random.choice(['A', 'B', 'C', 'D'], 100000)
})

mem = df.memory_usage(deep=True)
print('Memory per column (bytes):')
print(mem)
print(f'\nTotal: {mem.sum() / 1e6:.2f} MB')

memory_profiler for Line-by-Line Memory

The memory_profiler package (pip install memory-profiler) measures RAM usage line by line in a function. Decorate a function with @profile and run it with python -m memory_profiler script.py. In Jupyter, use the %memit magic (from the ipython extension) to measure peak memory of a single expression. This identifies which data structures cause sudden memory spikes — often concatenating in a loop instead of collecting and concatenating once.

# Install: pip install memory-profiler
# Usage as a decorator:

# from memory_profiler import profile
# @profile
# def process_data():
#     import pandas as pd, numpy as np
#     df = pd.DataFrame({'x': np.random.randn(1000000)})
#     result = df['x'].rolling(100).mean()    # <-- sees if this spikes RAM
#     return result

# Run: python -m memory_profiler script.py

# In Jupyter (after %load_ext memory_profiler):
# %memit pd.DataFrame({'x': np.random.randn(1000000)}).rolling(100).mean()

print('memory_profiler decorates functions for line-by-line RAM measurement.')
print('Typical output: Line # Mem usage Increment Line Contents')

sys.getsizeof vs memory_usage for Objects

sys.getsizeof(obj) returns the memory size of a Python object, but for pandas DataFrames it only reports the container overhead (a few hundred bytes), not the actual data. Always use df.memory_usage(deep=True).sum() for DataFrames. For individual Python objects like lists and dicts, sys.getsizeof is accurate but does not recurse into nested objects — use the pympler library (asizeof) for deep object sizes.

import sys
import pandas as pd
import numpy as np

df = pd.DataFrame({'x': np.random.randn(100000)})

# sys.getsizeof reports only the container metadata — misleading!
print('sys.getsizeof(df):', sys.getsizeof(df), 'bytes (misleading!)')

# memory_usage gives the true memory
true_mem = df.memory_usage(deep=True).sum()
print('df.memory_usage(deep=True).sum():', true_mem, 'bytes')
print(f'True memory: {true_mem/1e6:.2f} MB')

Timing Context Manager Pattern

For longer pipeline steps where timeit is not suitable (e.g. database queries or file I/O), use a simple context manager with time.perf_counter(). Wrap each pipeline step in a timer context to log its duration. This is more practical than timeit for production code because it logs real wall-clock time including I/O wait, which timeit also measures but is harder to integrate with logging frameworks.

import time
import contextlib
import pandas as pd
import numpy as np

@contextlib.contextmanager
def timer(name):
    start = time.perf_counter()
    yield
    elapsed = time.perf_counter() - start
    print(f'[{name}] {elapsed*1000:.1f} ms')

np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(500000), 'y': np.random.randn(500000)})

with timer('rolling mean'):
    result1 = df['x'].rolling(30).mean()

with timer('groupby mean'):
    df['group'] = df['x'].round(0)
    result2 = df.groupby('group')['y'].mean()

Profiling Pandas Operations Systematically

A systematic profiling workflow: 1) Start with a representative sample of your data (10,000 rows is enough for timing comparisons). 2) Identify the slowest steps using a timer context or cProfile. 3) Rewrite the slowest step using a faster approach (vectorisation, Categorical dtype, C extension). 4) Verify correctness with assert statements. 5) Re-time to confirm the speedup. 6) Repeat for the next bottleneck until the pipeline meets its SLA.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({
    'value': np.random.randn(100000),
    'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 100000)
})

# Benchmark two groupby implementations
t1 = timeit.timeit(lambda: df.groupby('category')['value'].mean(), number=100)
df2 = df.copy()
df2['category'] = df2['category'].astype('category')
t2 = timeit.timeit(lambda: df2.groupby('category')['value'].mean(), number=100)

print(f'Object dtype groupby: {t1/100*1000:.2f} ms per call')
print(f'Categorical dtype:    {t2/100*1000:.2f} ms per call')
print(f'Speedup: {t1/t2:.1f}x')

When to Profile: Practical Thresholds

Not every script needs profiling. Invest in profiling when: a pipeline takes more than 30 seconds to run and runs frequently (daily or more), a function processes more than 100,000 rows, or memory usage exceeds 50% of available RAM. For scripts that run once or in seconds, readability matters more than microsecond optimisation. The cost of maintaining heavily optimised, obscure code often exceeds the time saved by the optimisation.

Snapshot: Profiling Tools Summary

A quick reference:

  • timeit.timeit — measure time of a single statement (script)
  • %timeit / %%timeit — measure time in Jupyter (auto-repeats)
  • %prun / cProfile — function-level call stack profiling
  • df.memory_usage(deep=True) — per-column RAM measurement
  • memory_profiler @profile — line-by-line RAM measurement
  • time.perf_counter() — manual timer for pipeline steps
Always profile a representative data sample and confirm optimisations with assert checks before moving to production.

Quick Check

Test your understanding of profiling tools from this lesson.

Lesson Recap

In this lesson you learned: timeit.timeit and %timeit measure execution time of specific operations, df.memory_usage(deep=True) measures per-column RAM accurately, and cProfile / memory_profiler provide deeper profiling for complex pipelines. Always profile before optimising, and verify correctness after each change. Next up we explore how to avoid slow iterrows loops and replace them with vectorised operations.

常见问题解答

「使用 timeit 和 memory_profiler 进行性能分析」课时是免费的吗？

是的 — 「使用 timeit 和 memory_profiler 进行性能分析」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 Pandas & NumPy Academy 课程的其余内容，请升级到 CoddyKit PRO。 Pandas & NumPy Academy 课程共包含 4 节课。

「使用 timeit 和 memory_profiler 进行性能分析」这节课中我会学到什么？

使用 %timeit 测量执行时间，使用 memory_profiler 测量峰值内存，从而找出 pipeline 中最慢的部分。 你通过在浏览器中直接运行的动手代码来练习 Pandas & NumPy Academy，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 Pandas & NumPy Academy 需要有经验吗？

无需任何先前经验。CoddyKit 上的 Pandas & NumPy Academy 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 1 节课，共 4 节。

「使用 timeit 和 memory_profiler 进行性能分析」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 Pandas & NumPy Academy 课中编写并运行代码吗？

能。每节 Pandas & NumPy Academy 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 使用 timeit 和 memory_profiler 进行性能分析
  2. 避免使用 iterrows 和 Python 循环
  3. 用于减少内存的高效数据类型
  4. 大文件的分块读取
← 返回 Pandas & NumPy Academy