Parsing Dates Correctly
Parse date strings into datetime64 with pd.to_datetime, handle multiple formats, and set a DatetimeIndex.
Parsing Dates Correctly is a free Pandas & NumPy Academy lesson on CoddyKit — lesson 4 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Pandas & NumPy Academy learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Why Date Parsing Matters
Date and time data is ubiquitous in real-world datasets — transaction timestamps, birth dates, event logs, and financial periods. When dates are stored as strings (the object dtype), you cannot compute durations, resample by month, or sort chronologically. Converting strings to Pandas' datetime64 dtype unlocks the full time series API and makes temporal analysis possible.
import pandas as pd
df = pd.DataFrame({'date': ['2024-01-15', '2024-03-20', '2024-07-04']})
print(df['date'].dtype) # object
# You cannot do arithmetic on string dates
# df['date'] + pd.Timedelta('1 day') # TypeError!
# After parsing:
df['date'] = pd.to_datetime(df['date'])
print(df['date'].dtype) # datetime64[ns]
print(df['date'] + pd.Timedelta('1 day'))pd.to_datetime() Basics
pd.to_datetime(series) is the primary function for converting string columns to datetime64. It is smart enough to recognise most common date formats automatically (ISO 8601, US date, European date) without requiring a format string. The result is a Series with dtype datetime64[ns] — nanosecond precision, capable of storing timestamps from 1677 to 2262.
import pandas as pd
dates = pd.Series([
'2024-01-15',
'15/01/2024',
'January 15, 2024',
'2024-01-15 08:30:00'
])
parsed = pd.to_datetime(dates)
print(parsed)
# 0 2024-01-15 00:00:00
# 1 2024-01-15 00:00:00
# 2 2024-01-15 00:00:00
# 3 2024-01-15 08:30:00
print(parsed.dtype) # datetime64[ns]Specifying the format= Argument
When automatic inference is too slow or ambiguous (e.g., '01-02-03' could be January 2nd 2003 or February 1st 2003 or February 3rd 2001), provide the exact format string using Python's strftime codes. This is also significantly faster — on large datasets, explicit format parsing can be 10x faster than auto-inference because Pandas doesn't need to probe multiple patterns.
import pandas as pd
dates = pd.Series(['15/01/2024', '20/03/2024', '04/07/2024'])
# Explicit format: day/month/year
parsed = pd.to_datetime(dates, format='%d/%m/%Y')
print(parsed)
# 0 2024-01-15
# 1 2024-03-20
# 2 2024-07-04
# Common format codes:
# %Y = 4-digit year, %y = 2-digit year
# %m = month (01-12), %d = day (01-31)
# %H = hour (0-23), %M = minute, %S = seconderrors='coerce' for Bad Dates
Real datasets often contain typos or placeholder strings like 'TBD', 'N/A', or '99/99/9999' in date columns. Passing errors='coerce' tells pd.to_datetime() to convert unparseable values to NaT (Not a Time — the datetime equivalent of NaN) instead of raising an exception. This lets you identify and handle bad dates separately.
import pandas as pd
dates = pd.Series(['2024-01-15', 'TBD', '2024-07-04', 'N/A'])
parsed = pd.to_datetime(dates, errors='coerce')
print(parsed)
# 0 2024-01-15
# 1 NaT
# 2 2024-07-04
# 3 NaT
# Detect bad dates
print('Bad date rows:', parsed.isna().sum()) # 2parse_dates in read_csv()
The most efficient place to parse dates is at load time using the parse_dates parameter of pd.read_csv(). Pass a list of column names (or indices) that should be parsed as dates. This avoids a separate post-processing step and is often faster because Pandas handles it during the C-level CSV parsing.
import pandas as pd
import io
csv_data = '''order_id,order_date,ship_date
1,2024-01-15,2024-01-18
2,2024-02-20,2024-02-22
'''
df = pd.read_csv(
io.StringIO(csv_data),
parse_dates=['order_date', 'ship_date']
)
print(df.dtypes)
# order_id int64
# order_date datetime64[ns]
# ship_date datetime64[ns]The .dt Accessor for Date Components
Once a column has datetime64 dtype, the .dt accessor unlocks dozens of properties and methods. You can extract .dt.year, .dt.month, .dt.day, .dt.hour, .dt.dayofweek (0=Monday), and .dt.is_month_end. These are used for feature engineering in time series forecasting and for grouping data by time period.
import pandas as pd
df = pd.DataFrame({
'event_time': pd.to_datetime([
'2024-03-15 08:30:00',
'2024-07-04 14:00:00',
'2024-12-25 09:00:00'
])
})
df['year'] = df['event_time'].dt.year
df['month'] = df['event_time'].dt.month
df['day_of_week'] = df['event_time'].dt.day_name()
df['hour'] = df['event_time'].dt.hour
print(df[['year', 'month', 'day_of_week', 'hour']])Setting a DatetimeIndex
For time series analysis, it is standard to set the datetime column as the DataFrame's index. With a DatetimeIndex, you can use time-based slicing (df['2024'], df['2024-01':'2024-06']), resample to any frequency, and access Pandas' full time series API. Use set_index() or pass index_col in read_csv.
import pandas as pd
df = pd.DataFrame({
'date': pd.to_datetime(['2024-01-01', '2024-01-02', '2024-01-03']),
'temp': [22.5, 23.0, 21.8]
})
df = df.set_index('date')
print(df)
# temp
# date
# 2024-01-01 22.5
# 2024-01-02 23.0
# 2024-01-03 21.8
# Time-based slicing
print(df['2024-01-02':'2024-01-03'])Handling Multiple Date Formats
If a date column contains a mix of formats (e.g., some rows have ISO dates, others have US-style dates), you cannot specify a single format string. Pass format='mixed' (Pandas 2.0+) to allow row-by-row format detection, or pre-clean the column with regex to normalise all dates to one format before calling pd.to_datetime().
import pandas as pd
mixed_dates = pd.Series([
'2024-01-15',
'03/20/2024',
'2024-07-04T10:30:00'
])
# format='mixed' handles different formats row-by-row (Pandas >= 2.0)
parsed = pd.to_datetime(mixed_dates, format='mixed')
print(parsed)
# 0 2024-01-15 00:00:00
# 1 2024-03-20 00:00:00
# 2 2024-07-04 10:30:00Timezone-Aware Datetimes
Real-world timestamps often come with timezone information. pd.to_datetime() can parse UTC offset strings (e.g., '2024-01-15 08:30:00+05:30') and store them as timezone-aware datetime64. Use .dt.tz_convert('UTC') to normalise all timestamps to a common timezone for correct comparison and arithmetic.
import pandas as pd
dates = pd.to_datetime(['2024-01-15 08:30:00+05:30', '2024-01-15 10:00:00-08:00'])
print(dates)
# DatetimeIndex(['2024-01-15 08:30:00+05:30', '2024-01-15 10:00:00-08:00'])
# Convert both to UTC for fair comparison
utc_dates = dates.tz_convert('UTC')
print(utc_dates)
# DatetimeIndex(['2024-01-15 03:00:00+00:00', '2024-01-15 18:00:00+00:00'])Computing Date Differences
Arithmetic between two datetime64 columns produces a Timedelta Series. You can extract the number of days, hours, or seconds from a Timedelta using the .dt.days, .dt.seconds, and .dt.total_seconds() properties. This is the standard way to compute age, duration, or days-since-last-event features.
import pandas as pd
df = pd.DataFrame({
'start': pd.to_datetime(['2024-01-01', '2024-03-10', '2024-06-15']),
'end': pd.to_datetime(['2024-01-20', '2024-04-05', '2024-06-30'])
})
df['duration_days'] = (df['end'] - df['start']).dt.days
print(df[['start', 'end', 'duration_days']])
# start end duration_days
# 0 2024-01-01 2024-01-20 19
# 1 2024-03-10 2024-04-05 26
# 2 2024-06-15 2024-06-30 15pd.date_range() for Generating Dates
pd.date_range(start, end, freq=) generates a sequence of evenly spaced dates as a DatetimeIndex. This is useful for creating a complete time spine that you can reindex your data against — ensuring every calendar period appears in your output even if there were no events on that day. Common freq values: 'D' (daily), 'ME' (month-end), 'h' (hourly).
import pandas as pd
# Weekly dates for the first quarter of 2024
weekly = pd.date_range(start='2024-01-01', end='2024-03-31', freq='W')
print(weekly[:5])
# DatetimeIndex(['2024-01-07', '2024-01-14', '2024-01-21', '2024-01-28', '2024-02-04'])
# Monthly date spine
monthly = pd.date_range(start='2024-01', periods=12, freq='ME')
print(len(monthly), monthly[0], monthly[-1])
# 12 2024-01-31 2024-12-31Quick Check
Test your understanding of parsing dates correctly in Pandas.
Lesson Recap
In this lesson you learned: pd.to_datetime() converts string columns to datetime64, format= specifies the exact pattern for speed and accuracy, and errors='coerce' turns bad dates into NaT instead of crashing. The .dt accessor extracts year, month, day, and other components, and setting a DatetimeIndex enables time-based slicing. Next up we explore the .str accessor for vectorised string operations.
Frequently asked questions
Is the “Parsing Dates Correctly” lesson free?
Yes — the full text of “Parsing Dates Correctly” is free to read here on the web, and the Pandas & NumPy Academy course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Pandas & NumPy Academy course, upgrade to CoddyKit PRO.
What will I learn in “Parsing Dates Correctly”?
Parse date strings into datetime64 with pd.to_datetime, handle multiple formats, and set a DatetimeIndex. You practise Pandas & NumPy Academy with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Pandas & NumPy Academy?
No prior experience is required. Pandas & NumPy Academy on CoddyKit is structured for beginners through advanced learners; this is — lesson 4 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Parsing Dates Correctly” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Pandas & NumPy Academy lesson?
Yes. Every Pandas & NumPy Academy lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Inspecting Column Data Types
- Casting with astype()
- Categorical Data Type
- Parsing Dates Correctly