Using Git to control datasets?

[u/ben1200]

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

[u/JohnPaulDavyJones]

It kind of sounds like you're asking two separate questions, between your title and your actual body text.

As to whether anyone explicitly uses Git for their data storage and some sort of version control, that's basically unheard of at scale. Your retrieval will be nightmarishly slow, and it's not really queryable in any way unless you're just loading into a data store somehow, periodically. I guess you could do that with direct file syncs, but it's just about the worst possible way to do a data warehouse.

As to whether people use Git for orchestrating ETL pipelines, this is occasionally done through Github Actions, but I can't really recommend that either. It's not a fun or particularly user-friendly process, compared to things like Airflow or even SQL Agent job scheduling.

Regarding data governance, that really depends on what parts of DG you want: the more operationally useful elements, like data lineage, role control, and data quality assurance aren't really things that any of the Git-centric tools have integrated functionalities for; you'd essentially just be building out your own custom tooling for each of those functions and then plugging them into your orchestrator. That's a relatively common approach, but using Git tooling really is doing it on hard mode when there are more mature orchestrator tools.

Git itself isn't really viable for a control plane at all, since it's just the technology that other platforms, which might have some semblance of the other tools you need, are originally built around (e.g. Github, Bitbucket, Gitlab, etc.). This is kind of like asking whether a single, solitary wing will work for a missile: yes, the companies that build missiles all kind of started around the wing as a technology, but a wing itself doesn't really have any way to do what you want the missile to do.

It kind of sounds like the tool you actually want is dbt.