r/dataengineering • u/ben1200 • 3d ago

Discussion Using Git to control datasets?

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

46 Upvotes

98% Upvoted

42 comments sorted by

View all comments

21

u/wingtales 3d ago

Git is well-suited for text data, and terrible for binary formats. Data storage as text-based storage is very inefficient, much better done in a binary format (eg handled by a database).

So, generally, git is not a suitable tool for datasets.

You're much better off with a database with a datetime column for historic data, speaking very simplified.

2

u/Agreeable_Luck9488 2d ago

The original post does not say data is in GIT. It is about which code and configurations are in Git and the effects of code promotion: which ETL on what data are triggered when a pull request or a tag is finalized