为什么仅靠 Git 无法管理数据版本

Git was built for source code: small text files that change line by line. Your datasets are often gigabytes of binary data, a totally different beast.

Git keeps a full snapshot of each file version forever. Commit a 2 GB file ten times and your repo can balloon toward 20 GB of history. 😬

Git shines on text because it can show line diffs. With binary files like images or parquet, it cannot diff meaningfully and just stores the whole new copy.

Because history lives in the repo, a teammate cloning it must download every past dataset version too. A simple clone can take ages and fill the disk.

GitHub caps individual files at 100 MB and warns past 50 MB. Most real datasets blow right past that and the push is rejected.

Git LFS swaps big files for pointers, easing the size problem. But it lacks ML features like pipelines, remotes per project, and reproducible data stages.

An experiment is only trustworthy if you can recreate the exact data it used. Git alone cannot reliably link a commit to a specific dataset snapshot.

The fix is to keep a tiny pointer file in Git that names the data, while the heavy bytes live in cheap object storage. Git tracks the pointer, not the payload.

DVC (Data Version Control) does exactly this. It pairs Git for code with a separate cache and remote for data, giving you Git-like commands for datasets.

With DVC, checking out an old Git commit also restores the matching data version. Your code and dataset move together as one consistent unit.

DVC borrows Git's mental model: you add, commit, push, and checkout data. If you know Git basics, you already half-understand DVC. 🎉

Why does committing large datasets straight into Git cause trouble?

Git is great for code but chokes on big binary data, bloating history and hitting size limits. DVC stores tiny pointers in Git and keeps the heavy data elsewhere, in sync.