Shifting and Lag Features
Create lag and lead columns with shift(), compute period-over-period change with diff(), and calculate percentage change.
Shifting and Lag Features is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Lag Features Matter
In time series analysis, the value at a previous time step (a lag) is often one of the best predictors of the current value. For example, yesterday's sales are informative about today's sales. Creating lag columns lets you use historical values as features in machine learning models or in statistical analysis of autocorrelation. Pandas provides shift() to create lag features in a single vectorised call.
shift(): Moving Values Forward or Backward
Series.shift(n) moves all values down by n positions (positive n creates a lag), filling the first n rows with NaN. Passing a negative n moves values up (creates a lead, shifting future values back to the current row). The index remains unchanged — only the values are displaced.
import pandas as pd
import numpy as np
df = pd.DataFrame(
{'sales': [100, 120, 115, 130, 140, 125]},
index=pd.date_range('2024-01-01', periods=6, freq='D')
)
# Lag-1: yesterday's sales
df['sales_lag1'] = df['sales'].shift(1)
# Lead-1: tomorrow's sales (shifted up)
df['sales_lead1'] = df['sales'].shift(-1)
print(df)Creating Multiple Lag Columns
You typically create several lag features at once for machine learning — lag-1, lag-7 (same day last week), and lag-30 (same day last month). The most readable way is to create them in a loop and assign each to a new column. The resulting DataFrame has the original series plus all its lagged versions ready for modelling.
for lag in [1, 2, 3, 7]:
df[f'sales_lag{lag}'] = df['sales'].shift(lag)
print(df.columns.tolist())
# ['sales', 'sales_lag1', 'sales_lag2', 'sales_lag3', 'sales_lag7']
# Drop rows with NaN from the largest lag period
df_model = df.dropna()
print('Ready for modelling, shape:', df_model.shape)shift() with freq for Time-Based Shifting
Instead of shifting by an integer number of rows, you can shift by a time offset using the freq parameter. df.shift(1, freq='ME') shifts the index forward by one month-end, keeping the values in place but changing the index. This is useful for aligning two time series that are offset by a fixed time period without rearranging the values.
monthly = pd.Series(
[100, 120, 115],
index=pd.date_range('2024-01-31', periods=3, freq='ME')
)
# Shift the index forward by 1 month (values stay, index moves)
shifted_index = monthly.shift(1, freq='ME')
print('Original index:', monthly.index.tolist())
print('Shifted index: ', shifted_index.index.tolist())
# Original: [2024-01-31, 2024-02-29, 2024-03-31]
# Shifted: [2024-02-29, 2024-03-31, 2024-04-30]diff(): Period-over-Period Change
Series.diff(n) computes the difference between a value and the value n positions before it: x[t] - x[t-n]. This is commonly used to compute day-over-day changes, week-over-week differences, or year-over-year deltas. The first n rows are NaN since there are no previous values to subtract from.
df['sales_diff1'] = df['sales'].diff(1) # day-over-day change
df['sales_diff7'] = df['sales'].diff(7) # week-over-week change
print(df[['sales', 'sales_diff1']].head())
# sales sales_diff1
# 2024-01-01 100 NaN
# 2024-01-02 120 20.0 <- +20 from yesterday
# 2024-01-03 115 -5.0 <- -5 from yesterdaypct_change(): Percentage Change
Series.pct_change(n) computes the relative percentage change: (x[t] - x[t-n]) / x[t-n]. This is essential for financial analysis (daily returns), growth rate computation, and any situation where the absolute change is less meaningful than the relative change. Multiply by 100 to get percentage points.
df['pct_chg'] = df['sales'].pct_change().round(3)
df['pct_chg_7d'] = df['sales'].pct_change(7).round(3)
print(df[['sales', 'pct_chg']].head())
# sales pct_chg
# 2024-01-01 100 NaN
# 2024-01-02 120 0.200 <- 20% increase
# 2024-01-03 115 -0.042 <- 4.2% decreaseCombining shift() with Arithmetic
You can combine shift() with arithmetic to compute derived time features. For example: the ratio of today's value to last week's value, the cumulative sum since a fixed start, or the difference from the year-ago value. All of these follow the same pattern: shift the original series and perform element-wise arithmetic with the current values.
# Week-over-week growth index (today / last week)
df['wow_ratio'] = (df['sales'] / df['sales'].shift(7)).round(3)
# Deviation from 3-day lag
df['dev_3d'] = df['sales'] - df['sales'].shift(3)
# Is today higher than yesterday? (boolean feature)
df['higher_than_yesterday'] = df['sales'] > df['sales'].shift(1)
print(df[['sales', 'wow_ratio', 'higher_than_yesterday']].head())cumsum() and cumprod() for Cumulative Features
Series.cumsum() computes the running total from the first row up to each row, while Series.cumprod() computes the cumulative product. These are useful for cumulative revenue, cumulative returns (compound growth), and year-to-date totals. They require no arguments and work on any numeric Series or DataFrame column.
# Cumulative sales (year-to-date total)
df['ytd_sales'] = df['sales'].cumsum()
# Cumulative product: compound growth factor
returns = pd.Series([0.02, -0.01, 0.03, 0.01, -0.005])
df_ret = pd.DataFrame({'daily_return': returns})
df_ret['compound'] = (1 + df_ret['daily_return']).cumprod() - 1
print(df_ret)Lag Features for Machine Learning
When preparing time series data for machine learning, a standard feature engineering step is to create a matrix of lag features. Each column represents the target variable at a different past time step. After creating lags, drop rows with NaN (which appear at the start due to missing history) and split into train and test sets without shuffling, respecting the temporal order.
def make_lag_matrix(series, n_lags):
df_lags = pd.DataFrame({'y': series})
for lag in range(1, n_lags + 1):
df_lags[f'lag_{lag}'] = series.shift(lag)
return df_lags.dropna()
lag_df = make_lag_matrix(df['sales'], n_lags=3)
print(lag_df)
# y lag_1 lag_2 lag_3
# ... ... ... ...Shifting Within Groups
When your data contains multiple entities (e.g., multiple products or regions in one DataFrame), you must compute shifts within each entity's group separately. Use groupby().shift() to ensure that the lag for product A's first row is NaN and not the last value of product B. Mixing groups without this step is a common and subtle data leakage bug.
df_multi = pd.DataFrame({
'product': ['A', 'A', 'A', 'B', 'B', 'B'],
'sales': [100, 120, 115, 200, 210, 195]
})
# WRONG: lag crosses product boundary
df_multi['lag_wrong'] = df_multi['sales'].shift(1)
# CORRECT: lag within each product
df_multi['lag_correct'] = df_multi.groupby('product')['sales'].shift(1)
print(df_multi)Interpreting pct_change Signs
A positive pct_change() value means the current value is higher than the previous one; a negative value means it is lower. Watch out for division-by-zero when the previous value is zero — the result is NaN or inf depending on the sign of the current value. Use replace([float('inf'), float('-inf')], float('nan')) to clean infinite values after pct_change().
import numpy as np
s = pd.Series([0, 100, 50, 200])
chg = s.pct_change()
print(chg)
# 0 NaN <- no prior value
# 1 inf <- 0 -> 100 (division by zero)
# 2 -0.5 <- 100 -> 50 (-50%)
# 3 3.0 <- 50 -> 200 (+300%)
# Clean infinite values
chg_clean = chg.replace([float('inf'), float('-inf')], float('nan'))
print(chg_clean)Quick Check
Test your understanding of shifting and lag features from this lesson.
Lesson Recap
In this lesson you learned: shift(n) creates lag features by moving values down by n positions; diff(n) computes period-over-period absolute change; pct_change(n) computes relative percentage change; and when working with multiple groups you must use groupby().shift() to avoid data leakage. Next up we extract temporal features using the .dt accessor.
Frequently asked questions
Is the “Shifting and Lag Features” lesson free?
Yes — the full text of “Shifting and Lag Features” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Shifting and Lag Features”?
Create lag and lead columns with shift(), compute period-over-period change with diff(), and calculate percentage change. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Shifting and Lag Features” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- DatetimeIndex and Period Ranges
- Resampling Time Series
- Shifting and Lag Features
- Extracting Temporal Features