0Pricing
Pandas & NumPy Academy · Lesson

Avoiding iterrows and Python Loops

Replace row-by-row loops with vectorised column operations, np.where, and pd.cut to achieve 10-100x speedups.

Avoiding iterrows and Python Loops is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

The Cost of Row-by-Row Iteration

Pandas is built on NumPy, which processes entire arrays at once using compiled C code. When you iterate row by row with for _, row in df.iterrows(), you bypass this and fall back to pure Python — each row is extracted as a Series object, and the loop runs in the Python interpreter at roughly 100-1000x the cost of an equivalent vectorised operation. For a 1-million-row DataFrame, this can mean minutes instead of milliseconds.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(10000), 'b': np.random.randn(10000)})

# Slow: iterrows loop
def loop_version(df):
    result = []
    for _, row in df.iterrows():
        result.append(row['a'] + row['b'])
    return pd.Series(result)

# Fast: vectorised
t_loop = timeit.timeit(lambda: loop_version(df), number=5)
t_vec = timeit.timeit(lambda: df['a'] + df['b'], number=500)

print(f'Loop (5 runs):       {t_loop:.3f}s total')
print(f'Vectorised (500 runs): {t_vec:.3f}s total')
print(f'Speedup: ~{(t_loop/5)/(t_vec/500):.0f}x')

Replace Conditional Loops with np.where

np.where(condition, value_if_true, value_if_false) is the vectorised equivalent of an element-wise if/else. Instead of iterating rows and writing an if statement, pass the condition and both outcome values as arrays. np.where computes this in C for the entire column at once, making it 50-200x faster than an equivalent Python loop for large DataFrames.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({'price': np.random.uniform(10, 100, 100000)})

# Slow: iterrows
def label_loop(df):
    labels = []
    for _, row in df.iterrows():
        if row['price'] > 50:
            labels.append('expensive')
        else:
            labels.append('cheap')
    return labels

# Fast: np.where
def label_vectorised(df):
    return np.where(df['price'] > 50, 'expensive', 'cheap')

# Verify equivalence on a small subset
assert list(label_loop(df.head(100))) == list(label_vectorised(df.head(100)))
print('Results match!')
print('Fast version result sample:', label_vectorised(df)[:5])

Multiple Conditions with np.select

For three or more conditions, use np.select(condlist, choicelist, default=) instead of nested np.where. Pass a list of boolean arrays and a list of corresponding output values. The first matching condition determines the output; unmatched rows get the default value. This replaces complex if/elif/else chains inside loops with a clean, vectorised expression.

import pandas as pd
import numpy as np

np.random.seed(0)
df = pd.DataFrame({'score': np.random.randint(0, 101, 20)})

conditions = [
    df['score'] >= 90,
    df['score'] >= 75,
    df['score'] >= 60
]
choices = ['A', 'B', 'C']

df['grade'] = np.select(conditions, choices, default='F')
print(df.sort_values('score', ascending=False).head(10))

Vectorised String Operations via .str

String manipulation is a common source of slow loops. The Pandas .str accessor provides vectorised equivalents of all Python string methods: .str.lower(), .str.strip(), .str.replace(), .str.contains(), and many more. Using .str methods is 10-50x faster than iterating rows and calling Python string methods manually.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
names = ['Alice Smith', 'BOB JONES', '  carol  ', 'Dave Brown'] * 25000
df = pd.DataFrame({'name': names})

# Slow: loop
def clean_loop(df):
    return [n.strip().title() for n in df['name']]

# Fast: .str accessor
def clean_vectorised(df):
    return df['name'].str.strip().str.title()

t1 = timeit.timeit(lambda: clean_loop(df), number=10)
t2 = timeit.timeit(lambda: clean_vectorised(df), number=50)

print(f'Loop (10 runs): {t1:.3f}s')
print(f'.str (50 runs): {t2:.3f}s')
print(f'Speedup: {(t1/10)/(t2/50):.0f}x')
print('Sample:', clean_vectorised(df).head(4).tolist())

Using pd.cut and pd.qcut Instead of Loops

pd.cut() and pd.qcut() vectorise binning operations that are often written as loops with multiple if/elif conditions. If you find yourself writing 'if value < 18: age_group = child elif value < 65: age_group = adult' inside a row loop, replace it with a single pd.cut() call that processes the entire column at once in C-speed.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({'age': np.random.randint(0, 90, 100000)})

# Slow: loop with conditionals
def categorise_loop(df):
    result = []
    for age in df['age']:
        if age < 18:
            result.append('child')
        elif age < 65:
            result.append('adult')
        else:
            result.append('senior')
    return result

# Fast: pd.cut
def categorise_cut(df):
    return pd.cut(df['age'], bins=[0, 18, 65, 100],
                  labels=['child', 'adult', 'senior'],
                  right=False)

assert list(categorise_loop(df.head(5))) == list(categorise_cut(df.head(5)).astype(str))
print('Fast version sample:', categorise_cut(df).head(5).tolist())

apply() is Not Always the Answer

Many tutorials recommend replacing loops with .apply(func), but apply is still a Python-level loop internally — it is only marginally faster than iterrows and much slower than true vectorisation. Reserve apply for cases where no vectorised alternative exists (complex multi-column logic). If a built-in Pandas or NumPy function can express the operation, always prefer it over apply.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({'x': np.random.randn(100000)})

# These all do the same thing — compare performance
t1 = timeit.timeit(lambda: df['x'].apply(lambda v: v**2), number=20)
t2 = timeit.timeit(lambda: df['x'] ** 2, number=200)
t3 = timeit.timeit(lambda: np.square(df['x']), number=200)

print(f'apply(v**2): {t1/20*1000:.2f} ms per run')
print(f'** operator: {t2/200*1000:.2f} ms per run')
print(f'np.square(): {t3/200*1000:.2f} ms per run')

Replacing groupby Loops with agg and transform

A common pattern is iterating over groups manually: for name, group in df.groupby('cat'): do_something(group). This is slow for the same reasons iterrows is slow. Replace it with groupby().agg() for producing summary statistics, or groupby().transform() for broadcasting group-level statistics back to the original row positions.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({
    'cat': np.random.choice(['A','B','C','D'], 100000),
    'val': np.random.randn(100000)
})

# Slow: manual loop to add group mean
def slow_group_mean(df):
    means = {}
    for name, group in df.groupby('cat'):
        means[name] = group['val'].mean()
    return df['cat'].map(means)

# Fast: transform
def fast_group_mean(df):
    return df.groupby('cat')['val'].transform('mean')

t1 = timeit.timeit(lambda: slow_group_mean(df), number=10)
t2 = timeit.timeit(lambda: fast_group_mean(df), number=100)
print(f'Loop:       {t1/10*1000:.1f} ms')
print(f'transform:  {t2/100*1000:.1f} ms')
print(f'Speedup: {(t1/10)/(t2/100):.0f}x')

When iterrows Is Acceptable

Despite being slow, there are legitimate uses for iterrows and itertuples: printing or logging information from rows (performance is irrelevant), constructing API payloads where each row drives an HTTP call (the network latency dominates), and debugging specific rows with complex conditions. If you have fewer than 1,000 rows and the operation runs once, the difference between a loop and vectorisation is milliseconds — not worth the code complexity.

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'order_id': [101, 102, 103],
    'product': ['Widget', 'Gadget', 'Doohickey'],
    'amount': [29.99, 49.99, 9.99]
})

# Acceptable use: building a structured log (small data, one-time)
for _, row in df.iterrows():
    print(f'Order {row["order_id"]}: {row["product"]} — ${row["amount"]:.2f}')

itertuples Is Faster Than iterrows

If you must iterate rows in Python (because no vectorised equivalent exists), use itertuples() instead of iterrows(). It returns each row as a named tuple rather than a Pandas Series, avoiding the Series construction overhead. This makes it typically 5-10x faster than iterrows. Access values with attribute notation: row.column_name. Avoid if column names have spaces (not valid attribute names).

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({'a': np.random.randn(10000), 'b': np.random.randn(10000)})

t1 = timeit.timeit(
    lambda: [row.a + row.b for row in df.itertuples()], number=10)
t2 = timeit.timeit(
    lambda: [row['a'] + row['b'] for _, row in df.iterrows()], number=10)

print(f'itertuples: {t1/10*1000:.1f} ms')
print(f'iterrows:   {t2/10*1000:.1f} ms')
print(f'itertuples speedup: {t2/t1:.1f}x')

Vectorisation Pattern Checklist

Before writing a loop, ask these questions:

  • Is the operation element-wise on one column? → Use a column arithmetic expression or NumPy ufunc.
  • Does it involve conditional logic? → Use np.where (2 conditions) or np.select (3+).
  • Does it bin values into ranges? → Use pd.cut or pd.qcut.
  • Does it apply string operations? → Use the .str accessor.
  • Does it aggregate within groups? → Use groupby().agg() or groupby().transform().
Only fall back to apply() or itertuples() if none of the above applies.

Real Speedup Example: Price Calculation

Here is a complete before/after refactoring of a common business calculation — applying tiered discounts based on order quantity. The loop version is readable but unacceptably slow for large DataFrames. The vectorised version using np.select achieves the same result in one tenth of the time.

import pandas as pd
import numpy as np
import timeit

np.random.seed(0)
df = pd.DataFrame({
    'price': np.random.uniform(10, 200, 100000),
    'qty': np.random.randint(1, 500, 100000)
})

# BEFORE: loop with conditionals
def slow_discount(df):
    result = []
    for _, row in df.iterrows():
        if row['qty'] >= 100:
            disc = 0.2
        elif row['qty'] >= 50:
            disc = 0.1
        elif row['qty'] >= 10:
            disc = 0.05
        else:
            disc = 0.0
        result.append(row['price'] * (1 - disc))
    return pd.Series(result)

# AFTER: np.select
def fast_discount(df):
    conditions = [df['qty'] >= 100, df['qty'] >= 50, df['qty'] >= 10]
    discounts = [0.20, 0.10, 0.05]
    disc = np.select(conditions, discounts, default=0.0)
    return df['price'] * (1 - disc)

assert slow_discount(df.head(200)).round(4).equals(fast_discount(df.head(200)).reset_index(drop=True).round(4))
print('Both versions agree!')

Quick Check

Test your understanding of vectorisation from this lesson.

Lesson Recap

In this lesson you learned: iterrows is 100-1000x slower than vectorised operations, np.where and np.select replace conditional loops, the .str accessor vectorises string operations, and groupby().transform() replaces manual group iteration. When you must iterate, use itertuples over iterrows for a 5-10x improvement. Next up we explore efficient data types — downcasting numerics and using Categorical to reduce memory by up to 70%.

Frequently asked questions

Is the “Avoiding iterrows and Python Loops” lesson free?

Yes — the full text of “Avoiding iterrows and Python Loops” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.

What will I learn in “Avoiding iterrows and Python Loops”?

Replace row-by-row loops with vectorised column operations, np.where, and pd.cut to achieve 10-100x speedups. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Pandas & NumPy Academy?

No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Avoiding iterrows and Python Loops” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Pandas & NumPy Academy lesson?

Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Profiling with timeit and memory_profiler
  2. Avoiding iterrows and Python Loops
  3. Efficient Data Types for Memory Reduction
  4. Chunked Reading for Large Files
← Back to Pandas & NumPy Academy