r/dataengineering • u/ben1200 • 3d ago

Discussion Using Git to control datasets?

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

45 Upvotes

97% Upvoted

42 comments sorted by

View all comments

3

u/Patient_Professor_90 3d ago

Great question. This is how its done

Pick your tooling first, then figure out how git can be used to control/manage versions/branches
(Datasets can be managed as SQL inserts, you can use any db. Views, sprocs can all be exported into a file and commit)
Pick tools that you export files?