Date and Time Feature Extraction

A raw Unix timestamp like 1718784000 or a datetime string like '2024-06-19 14:30:00' contains rich information — the hour of day, day of week, month of year — but a model cannot access that information from the raw value. A linear model sees the timestamp as a single large integer and can only learn that later timestamps predict higher/lower targets, missing all periodic patterns. Feature extraction decomposes the timestamp into meaningful numeric components that models can directly use.

Always parse date/time strings to pd.Timestamp or datetime64 dtype using pd.to_datetime() before extracting features. Once parsed, Pandas provides a .dt accessor with dozens of attributes: .dt.year, .dt.month, .dt.day, .dt.hour, .dt.minute, .dt.dayofweek (0=Monday), .dt.dayofyear, .dt.quarter, .dt.is_weekend, and more. Each attribute becomes a new integer column in your feature matrix.

Hours 23 and 0 are adjacent (one hour apart) but numerically far apart (23 units). If you encode hour as an integer 0-23, a linear model cannot learn that midnight patterns are similar to 11pm patterns. The same applies to months (December and January are adjacent), day-of-week (Sunday=6 and Monday=0), and any other cyclical quantity. Treating these as raw integers creates an artificial discontinuity at the cycle boundary that models have to work around.

The solution is to encode cyclical features as sine and cosine pairs. For a variable ranging from 0 to T-1, encode it as: sin(2π × value / T) and cos(2π × value / T). This maps the cycle onto a circle in 2D space. Hour 23 and hour 0 are now adjacent points on the circle, so their Euclidean distance is small. Any model that computes distances or linear combinations can now capture cyclical patterns correctly using both the sin and cos components together.

The cyclical encoding pattern generalises to any periodic feature: month (T=12), day of week (T=7), hour (T=24), minute (T=60), day of year (T=365). Each cyclical feature generates two new features (sin and cos). For tree-based models, raw integer encoding often works fine because trees split on thresholds and can capture cyclical patterns through multiple splits. But for linear models, neural networks, and distance-based algorithms, cyclical encoding is important.

Instead of extracting calendar components, sometimes what matters is time elapsed since an event: days since last purchase, hours since system restart, months since account creation. These relative duration features capture recency effects that absolute calendar features miss. Compute them as simple differences: (now - event_date).dt.days or (now - event_date).dt.total_seconds(). Recency features are especially valuable in customer behaviour models (churn, LTV) and predictive maintenance.

Boolean indicator features derived from timestamps are simple but often powerful. is_weekend, is_business_hours, is_holiday, is_month_end — these create clear binary splits that tree models can exploit in a single split. Linear models also benefit because these flags create distinct intercept adjustments. Adding a public holiday flag can be especially impactful for retail sales, traffic, and energy consumption models where holidays cause systematic demand shifts.

For time-series data, the past values of the target variable are often among the best predictors. Lag features shift the target by N timesteps: df['target_lag1'] = df['target'].shift(1) creates a feature equal to yesterday's target. Rolling statistics summarise recent history: df['rolling_mean_7'] = df['target'].rolling(7).mean() gives the 7-day moving average. These features encode temporal momentum and seasonality that calendar features alone cannot capture.

To use date feature extraction inside a scikit-learn Pipeline without leakage, create a custom transformer using BaseEstimator and TransformerMixin. This ensures date features are computed consistently and reproducibly within each fold of cross-validation. For lag and rolling features, be extra careful — rolling statistics computed across the full dataset (including future test data) will leak information. Always compute lag/rolling features only up to the training cutoff date.

Effective datetime feature engineering requires understanding which temporal patterns matter for your problem. For ride-sharing demand: hour-of-day and day-of-week are critical. For retail sales: day-of-week, week-of-year, proximity to holidays, and year-over-year trend. For network intrusion detection: time-since-last-event and burst rate. For energy consumption: hour, day, and temperature interactions. Always start by plotting the target against each temporal component to identify visible periodic patterns before engineering features.

When your dataset spans multiple time zones, always convert timestamps to a single reference timezone before feature extraction. Using UTC as the canonical timezone is standard practice. After extracting features, you can re-apply local-time offsets if local patterns matter (e.g., hour of day in customer's local time is more predictive than UTC hour for user behaviour modelling). The Pandas dt.tz_localize() and dt.tz_convert() methods handle timezone conversion cleanly.

Test your understanding of Date and Time feature extraction from this lesson.

In this lesson you learned: raw timestamps must be decomposed into meaningful components before use as features, cyclical features like hour and month require sine/cosine encoding to avoid boundary discontinuities, and lag and rolling features encode temporal patterns for time-series prediction tasks. Next up we explore Feature Selection with Variance Threshold and SelectKBest to discard uninformative features.