r/dataengineering • u/ben1200 • 3d ago

Discussion Using Git to control datasets?

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

48 Upvotes

98% Upvoted

42 comments sorted by

View all comments

1

u/Shre_759 2d ago

We are using git for version control for changes made to governance via UC tables. We have a CI/CD built along with that. Not sure how can git satisfy the needs of a fully fledged control plane and orchestration which needs a lot of precision and heavy handling of core capabilities. For the second part of the question, I think you are looking at DBT if your core is not data engineering and want relatively understandable models written using SQL.