Storing Collected Data Efficiently
Saving to CSV/JSON/Parquet, appending safely, deduplication, incremental collection.
Storing Collected Data Efficiently is a free Learn AI with Python lesson on CoddyKit — lesson 3 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Learn AI with Python learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
From Raw Responses to Stored Data
After collecting data you need to store it so it can be reloaded, shared, and analyzed. The right format and a few habits make a big difference in speed and disk usage.
This lesson covers CSV, Parquet, appending batches, and deduplication.
JSON to DataFrame
API responses are usually lists of dicts. pd.DataFrame turns that directly into a table ready for storage.
import pandas as pd
records = [
{"id": 1, "name": "A", "score": 9.5},
{"id": 2, "name": "B", "score": 8.0},
]
df = pd.DataFrame(records)
print(df)Saving to CSV with df.to_csv
CSV is universal and human-readable. Save with to_csv; pass index=False so the row index is not written as a stray column.
df.to_csv("data.csv", index=False)
# Reload
df2 = pd.read_csv("data.csv")Limits of CSV
CSV is convenient but has drawbacks for AI work:
- No types — everything is stored as text, so dtypes are re-guessed on load
- Large files, no compression by default
- Slow to read for big datasets
For larger or repeatedly-loaded data, Parquet is better.
Saving to Parquet with df.to_parquet
Parquet is a columnar binary format. It preserves dtypes, compresses well, and loads far faster than CSV.
It needs an engine such as pyarrow installed.
df.to_parquet("data.parquet", index=False)
df2 = pd.read_parquet("data.parquet")
print(df2.dtypes) # types preserved, no re-guessingCSV vs Parquet — When to Use Which
Rules of thumb:
- CSV: small data, sharing with non-Python tools, quick inspection
- Parquet: large data, repeated loads, type fidelity, analytics pipelines
For collection jobs that feed models, prefer Parquet.
Appending New Batches with pd.concat
Collection often happens in batches. Combine an existing DataFrame with a new one using pd.concat.
ignore_index=True renumbers the index so it stays clean.
new_batch = pd.DataFrame(new_records)
df = pd.concat([df, new_batch], ignore_index=True)
print(len(df))Appending Directly to a CSV File
To append to an existing CSV without loading it, open in append mode and skip the header on later writes.
import os
header = not os.path.exists("data.csv")
new_batch.to_csv("data.csv", mode="a", header=header, index=False)Why Deduplicate
Re-running collection, overlapping pages, or retries can create duplicate rows. Duplicates inflate counts and can leak between train/test splits, hurting model evaluation.
Always deduplicate after combining batches.
drop_duplicates(subset=[id])
Use a stable unique key (often id) with drop_duplicates(subset=...). This removes repeats even if other fields changed slightly.
keep="last" keeps the most recent version of each id.
df = df.drop_duplicates(subset=["id"], keep="last").reset_index(drop=True)
print(len(df), "unique rows")A Save-and-Merge Helper
Combine the pieces: load existing Parquet if present, concat the new batch, deduplicate by id, and save back. Safe to run repeatedly.
import os
import pandas as pd
def save_merge(new_df, path="data.parquet", key="id"):
if os.path.exists(path):
old = pd.read_parquet(path)
new_df = pd.concat([old, new_df], ignore_index=True)
new_df = new_df.drop_duplicates(subset=[key], keep="last")
new_df.to_parquet(path, index=False)
return new_dfQuick Check: Format Choice
You repeatedly load a large dataset for model training and need dtypes preserved with fast reads.
Recap: Storing Data Efficiently
You can now persist collected data well:
pd.DataFrameto turn JSON records into a tableto_csv(index=False)for portable text,to_parquetfor fast typed storagepd.concat(ignore_index=True)or CSV append mode for batchesdrop_duplicates(subset=["id"])to keep rows unique
Next: working with real public data APIs.
Frequently asked questions
Is the “Storing Collected Data Efficiently” lesson free?
Yes — the full text of “Storing Collected Data Efficiently” is free to read here on the web, and the Learn AI with Python course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Learn AI with Python course, upgrade to CoddyKit PRO.
What will I learn in “Storing Collected Data Efficiently”?
Saving to CSV/JSON/Parquet, appending safely, deduplication, incremental collection. You practise Learn AI with Python with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Learn AI with Python?
No prior experience is required. Learn AI with Python on CoddyKit is structured for beginners through advanced learners; this is — lesson 3 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Storing Collected Data Efficiently” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Learn AI with Python lesson?
Yes. Every Learn AI with Python lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- REST API Fundamentals for Data Collection
- Paginating and Collecting Large Datasets
- Storing Collected Data Efficiently
- Working with Public Data APIs