Pandas & NumPy Academy · 강의

timeit과 memory_profiler로 프로파일링하기

%timeit으로 실행 시간을, memory_profiler로 최대 메모리를 측정해 pipeline에서 가장 느린 부분을 찾습니다.

timeit과 memory_profiler로 프로파일링하기은(는) CoddyKit의 무료 Pandas & NumPy Academy 강의입니다. 이것은 4개 중 1번째 강의입니다. 아래에서 전체 강의를 무료로 읽을 수 있으며, 내장 코드 에디터와 24/7 AI 튜터와 함께 브라우저에서 직접 실습할 수 있습니다. 이 강의는 Pandas & NumPy Academy 학습 경로의 일부이며, 진행 상황이 웹과 CoddyKit 앱에 동기화됩니다. Pandas & NumPy Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

이 강의의 일부는 아직 번역되지 않았으며 영어로 표시됩니다.

Why Profile Before Optimising?

A common mistake is to optimise code intuitively — rewriting a function that looks slow without measuring whether it is actually the bottleneck. In data pipelines, 90% of the execution time often comes from 10% of the code. Profiling identifies exactly where time and memory are consumed so you can focus optimisation effort where it matters. The golden rule: measure first, optimise second. Tools like timeit measure time and memory_profiler measures RAM.

import pandas as pd
import numpy as np
import timeit

# Naive approach vs vectorised approach — which is faster?
np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(100000), 'b': np.random.randn(100000)})

t1 = timeit.timeit(lambda: df['a'] + df['b'], number=100)
t2 = timeit.timeit(lambda: [a + b for a, b in zip(df['a'], df['b'])], number=10)

print(f'Vectorised (100 runs): {t1:.3f}s')
print(f'List comprehension (10 runs): {t2:.3f}s')
print(f'Per-run speedup: ~{(t2/10)/(t1/100):.0f}x')

timeit: Python's Built-In Timer

The timeit module measures execution time by running a statement many times and returning the total elapsed time. timeit.timeit(stmt, number=n) runs stmt n times and returns total seconds. Divide by number to get per-run time. For accurate measurements, choose number so the total run takes at least 0.1 seconds — a single run of a fast operation is not representative due to system scheduling noise.

import timeit
import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(10000)})

# Compare three ways to square a column
time_a = timeit.timeit(lambda: df['x'] ** 2, number=1000)
time_b = timeit.timeit(lambda: df['x'].apply(lambda v: v**2), number=100)
time_c = timeit.timeit(lambda: np.square(df['x']), number=1000)

print(f'Operator **2  per run: {time_a/1000*1000:.3f} ms')
print(f'.apply(v**2) per run: {time_b/100*1000:.3f} ms')
print(f'np.square()  per run: {time_c/1000*1000:.3f} ms')

%timeit Magic in Jupyter

In Jupyter notebooks, the %timeit magic automatically selects the number of repetitions and runs to give a statistically stable estimate, then reports the best and mean times with standard deviation. It is the most convenient way to quickly benchmark a single line. Use %%timeit (double percent) to time a whole cell. The output format is: X ns/μs/ms ± Y ns per loop (mean ± std. dev. of 7 runs, N loops each).

# In Jupyter notebook:
# %timeit df['x'] ** 2
# Output: 47.3 us +- 1.2 us per loop (mean +- std. dev. of 7 runs, 10000 loops each)

# Multi-line cell timing:
# %%timeit
# result = df['x'] ** 2
# total = result.sum()

# In a regular Python script, use timeit.timeit() instead:
import timeit
import pandas as pd, numpy as np
df = pd.DataFrame({'x': np.random.randn(10000)})
result = timeit.repeat(lambda: df['x']**2, number=1000, repeat=7)
print(f'Best run: {min(result)/1000*1000:.3f} ms per call')

cProfile for Function-Level Profiling

While timeit measures a single expression, cProfile profiles the entire call stack — it shows how much time is spent in every function called during execution. This is essential for identifying the bottleneck in a complex pipeline. Run python -m cProfile -s cumulative script.py from the command line, or use cProfile.run('function()') in a script. In Jupyter, use %prun function().

import cProfile
import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(50000), 'b': np.random.randn(50000)})

def slow_pipeline(df):
    result = []
    for _, row in df.iterrows():
        result.append(row['a'] * row['b'])
    return pd.Series(result)

# Profile the function
print('Profile output (top 5 functions by cumulative time):')
cProfile.run('slow_pipeline(df.head(1000))', sort='cumulative')

Measuring Memory Usage

Use df.memory_usage(deep=True) to see how much RAM each column consumes. The deep=True argument measures the actual memory used by object dtype columns (strings), which otherwise reports only the pointer size. Use .sum() to get the total DataFrame memory in bytes. Divide by 1e6 for MB or 1e9 for GB. This is the starting point for memory optimisation — you cannot reduce what you have not measured.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({
    'id': range(100000),
    'name': ['user_' + str(i) for i in range(100000)],
    'score': np.random.randn(100000),
    'category': np.random.choice(['A', 'B', 'C', 'D'], 100000)
})

mem = df.memory_usage(deep=True)
print('Memory per column (bytes):')
print(mem)
print(f'\nTotal: {mem.sum() / 1e6:.2f} MB')

memory_profiler for Line-by-Line Memory

The memory_profiler package (pip install memory-profiler) measures RAM usage line by line in a function. Decorate a function with @profile and run it with python -m memory_profiler script.py. In Jupyter, use the %memit magic (from the ipython extension) to measure peak memory of a single expression. This identifies which data structures cause sudden memory spikes — often concatenating in a loop instead of collecting and concatenating once.

# Install: pip install memory-profiler
# Usage as a decorator:

# from memory_profiler import profile
# @profile
# def process_data():
#     import pandas as pd, numpy as np
#     df = pd.DataFrame({'x': np.random.randn(1000000)})
#     result = df['x'].rolling(100).mean()    # <-- sees if this spikes RAM
#     return result

# Run: python -m memory_profiler script.py

# In Jupyter (after %load_ext memory_profiler):
# %memit pd.DataFrame({'x': np.random.randn(1000000)}).rolling(100).mean()

print('memory_profiler decorates functions for line-by-line RAM measurement.')
print('Typical output: Line # Mem usage Increment Line Contents')

sys.getsizeof vs memory_usage for Objects

sys.getsizeof(obj) returns the memory size of a Python object, but for pandas DataFrames it only reports the container overhead (a few hundred bytes), not the actual data. Always use df.memory_usage(deep=True).sum() for DataFrames. For individual Python objects like lists and dicts, sys.getsizeof is accurate but does not recurse into nested objects — use the pympler library (asizeof) for deep object sizes.

import sys
import pandas as pd
import numpy as np

df = pd.DataFrame({'x': np.random.randn(100000)})

# sys.getsizeof reports only the container metadata — misleading!
print('sys.getsizeof(df):', sys.getsizeof(df), 'bytes (misleading!)')

# memory_usage gives the true memory
true_mem = df.memory_usage(deep=True).sum()
print('df.memory_usage(deep=True).sum():', true_mem, 'bytes')
print(f'True memory: {true_mem/1e6:.2f} MB')

Timing Context Manager Pattern

For longer pipeline steps where timeit is not suitable (e.g. database queries or file I/O), use a simple context manager with time.perf_counter(). Wrap each pipeline step in a timer context to log its duration. This is more practical than timeit for production code because it logs real wall-clock time including I/O wait, which timeit also measures but is harder to integrate with logging frameworks.

import time
import contextlib
import pandas as pd
import numpy as np

@contextlib.contextmanager
def timer(name):
    start = time.perf_counter()
    yield
    elapsed = time.perf_counter() - start
    print(f'[{name}] {elapsed*1000:.1f} ms')

np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(500000), 'y': np.random.randn(500000)})

with timer('rolling mean'):
    result1 = df['x'].rolling(30).mean()

with timer('groupby mean'):
    df['group'] = df['x'].round(0)
    result2 = df.groupby('group')['y'].mean()

Profiling Pandas Operations Systematically

A systematic profiling workflow: 1) Start with a representative sample of your data (10,000 rows is enough for timing comparisons). 2) Identify the slowest steps using a timer context or cProfile. 3) Rewrite the slowest step using a faster approach (vectorisation, Categorical dtype, C extension). 4) Verify correctness with assert statements. 5) Re-time to confirm the speedup. 6) Repeat for the next bottleneck until the pipeline meets its SLA.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({
    'value': np.random.randn(100000),
    'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 100000)
})

# Benchmark two groupby implementations
t1 = timeit.timeit(lambda: df.groupby('category')['value'].mean(), number=100)
df2 = df.copy()
df2['category'] = df2['category'].astype('category')
t2 = timeit.timeit(lambda: df2.groupby('category')['value'].mean(), number=100)

print(f'Object dtype groupby: {t1/100*1000:.2f} ms per call')
print(f'Categorical dtype:    {t2/100*1000:.2f} ms per call')
print(f'Speedup: {t1/t2:.1f}x')

When to Profile: Practical Thresholds

Not every script needs profiling. Invest in profiling when: a pipeline takes more than 30 seconds to run and runs frequently (daily or more), a function processes more than 100,000 rows, or memory usage exceeds 50% of available RAM. For scripts that run once or in seconds, readability matters more than microsecond optimisation. The cost of maintaining heavily optimised, obscure code often exceeds the time saved by the optimisation.

Snapshot: Profiling Tools Summary

A quick reference:

timeit.timeit — measure time of a single statement (script)
%timeit / %%timeit — measure time in Jupyter (auto-repeats)
%prun / cProfile — function-level call stack profiling
df.memory_usage(deep=True) — per-column RAM measurement
memory_profiler @profile — line-by-line RAM measurement
time.perf_counter() — manual timer for pipeline steps

Always profile a representative data sample and confirm optimisations with assert checks before moving to production.

Quick Check

Test your understanding of profiling tools from this lesson.

Lesson Recap

In this lesson you learned: timeit.timeit and %timeit measure execution time of specific operations, df.memory_usage(deep=True) measures per-column RAM accurately, and cProfile / memory_profiler provide deeper profiling for complex pipelines. Always profile before optimising, and verify correctness after each change. Next up we explore how to avoid slow iterrows loops and replace them with vectorised operations.

자주 묻는 질문

“timeit과 memory_profiler로 프로파일링하기” 강의는 무료인가요?

네 — “timeit과 memory_profiler로 프로파일링하기” 전체 내용을 이 웹사이트에서 무료로 읽을 수 있습니다. 인터랙티브하게 실습하려면(내장 코드 에디터와 24/7 AI 튜터), CoddyKit PRO로 업그레이드하면 Pandas & NumPy Academy 강의 전체를 잠금 해제할 수 있습니다. Pandas & NumPy Academy 강의에는 총 4개의 강의가 포함되어 있습니다.

“timeit과 memory_profiler로 프로파일링하기”에서 뭘 배우나요?

%timeit으로 실행 시간을, memory_profiler로 최대 메모리를 측정해 pipeline에서 가장 느린 부분을 찾습니다. 브라우저에서 직접 실행하는 실습 코드로 Pandas & NumPy Academy을(를) 배우며, 24/7 AI 튜터가 강의를 진행하면서 질문에 답변해줍니다.

Pandas & NumPy Academy을(를) 시작하는 데 경험이 필요한가요?

사전 경험은 필요하지 않습니다. CoddyKit의 Pandas & NumPy Academy은(는) 초급자부터 고급 학습자까지를 위해 구성되어 있으므로, 여기서 시작하거나 처음부터 시작할 수 있으며 자신의 속도대로 진행할 수 있습니다. 이것은 4개 중 1번째 강의입니다.

“timeit과 memory_profiler로 프로파일링하기” 강의는 얼마나 걸리나요?

대부분의 CoddyKit 강의는 약 5~10분이 소요됩니다. 각 강의는 간결하고 인터랙티브하여 꾸준한 진행이 가능하며, 웹과 앱에서 중단한 부분부터 바로 시작할 수 있습니다.

이 Pandas & NumPy Academy 강의에서 코드를 작성하고 실행할 수 있나요?

네. 모든 Pandas & NumPy Academy 강의에는 내장 코드 에디터가 포함되어 있으므로, 브라우저에서 바로 실제 코드를 작성하고 실행한 후 즉시 AI 피드백을 받을 수 있습니다 — 로컬 설정이 필요 없습니다.

이 강의의 모든 강의

← Pandas & NumPy Academy(으)로 돌아가기