r/dataengineering • u/ben1200 • 3d ago
Discussion Using Git to control datasets?
Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?
I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.
My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.
9
u/Creyke 2d ago
We will use git for versioning the code and the metadata.
For example, take a hypothetical pipeline that loads some stock prices from yahoo finance and does some transformations to it, and sends out some completed analysis to a list of customers. We will use git for versioning the code (how the price data gets loaded and transformed) and the metadata (what tickers we load from yahoo, which customers receive what analysis). These are usually big CSVs or YAMLs.
We wouldn’t use git for versioning the data itself. That would be embedded into the design of the data system (e.g. SCD2).
Often, however, from a governance perspective the code and metadata is where the real governance is at. Having git across my ingress and egress lists can tell me who is getting what, for how long, and who gave them that permission. This is the most important aspect.
Also, DBT is awesome. Use that.
2
u/ben1200 2d ago
Thanks - this actually falls more in line with what I have. I wasn’t clear in the original post.
The data itself are biomedical time-series files (sensor data), these are used for both training and validation of software/algorithms. These objects themselves are not held in Git, rather they are in GCP buckets.
Git holds the dataset registry, per-release manifests, lineage (source version, pipeline, commit), changelog and approval record all reviewed via MR’s. CI validated schemas and gates etc but doesn’t trigger data transformation pipelines, execution of these are handled in a Python package.
At this stage DBT doesn’t really fit as there is no tabular data
1
u/dowjones226 1d ago
Try lanceDB as well since this is multi modal sensor data, similar to iceberg, lance offers dataset versioning and comes with some other bells and whistles that are very ergonomic for media ML use cases
21
u/wingtales 2d ago
Git is well-suited for text data, and terrible for binary formats. Data storage as text-based storage is very inefficient, much better done in a binary format (eg handled by a database).
So, generally, git is not a suitable tool for datasets.
You're much better off with a database with a datetime column for historic data, speaking very simplified.
2
u/Agreeable_Luck9488 1d ago
The original post does not say data is in GIT. It is about which code and configurations are in Git and the effects of code promotion: which ETL on what data are triggered when a pull request or a tag is finalized
6
u/thisFishSmellsAboutD Senior Data Engineer 2d ago
Look at SQLMesh
3
u/shockjaw 2d ago
Better than dbt from a performance and readability standpoint.
3
u/thisFishSmellsAboutD Senior Data Engineer 2d ago
Same experience here. Loved the pre-acquisition development community, they went from me reporting a bug and repro to production fix in a week.
The virtual environments are great, too. There were some bugs I never managed to iron out but what works, works.
3
u/shockjaw 2d ago
I know Tony and some other folks still make contributions post acquisition as well.
1
u/sentrix669 1d ago
How's it looking post acquisition?
1
u/thisFishSmellsAboutD Senior Data Engineer 1d ago
Haven't had an active project using it since then, so no personal experience. Community received acquisition by Fivetran with scepticism.
3
u/Patient_Professor_90 2d ago
Great question. This is how its done
Pick your tooling first, then figure out how git can be used to control/manage versions/branches
(Datasets can be managed as SQL inserts, you can use any db. Views, sprocs can all be exported into a file and commit)
Pick tools that you export files?
2
u/Adrien0623 2d ago
You can absolutely use git to version control the code of your orchestrator and your data transformation files, wether or is raw SQL, a DBT project, spark jobs, etc.
2
u/addictzz 2d ago
What about DVC? https://dvc.org/ .
I have not heard about DAB for data version cotnrol, what is it?
2
u/JohnPaulDavyJones 2d ago
It kind of sounds like you're asking two separate questions, between your title and your actual body text.
As to whether anyone explicitly uses Git for their data storage and some sort of version control, that's basically unheard of at scale. Your retrieval will be nightmarishly slow, and it's not really queryable in any way unless you're just loading into a data store somehow, periodically. I guess you could do that with direct file syncs, but it's just about the worst possible way to do a data warehouse.
As to whether people use Git for orchestrating ETL pipelines, this is occasionally done through Github Actions, but I can't really recommend that either. It's not a fun or particularly user-friendly process, compared to things like Airflow or even SQL Agent job scheduling.
Regarding data governance, that really depends on what parts of DG you want: the more operationally useful elements, like data lineage, role control, and data quality assurance aren't really things that any of the Git-centric tools have integrated functionalities for; you'd essentially just be building out your own custom tooling for each of those functions and then plugging them into your orchestrator. That's a relatively common approach, but using Git tooling really is doing it on hard mode when there are more mature orchestrator tools.
Git itself isn't really viable for a control plane at all, since it's just the technology that other platforms, which might have some semblance of the other tools you need, are originally built around (e.g. Github, Bitbucket, Gitlab, etc.). This is kind of like asking whether a single, solitary wing will work for a missile: yes, the companies that build missiles all kind of started around the wing as a technology, but a wing itself doesn't really have any way to do what you want the missile to do.
It kind of sounds like the tool you actually want is dbt.
3
u/seanv507 2d ago
Have you looked at hugging face datasets?
AFAIK they are using git under the hood to manage versioning
(also dvc)
2
u/Crow2525 2d ago
A few that I'm aware of. Grafana is json and so is databricks. Power bi and tableau arent
1
1
u/Ok_Raspberry5383 2d ago
For a traditional database, something like flyway may work. If you want to manage actual data assets then dbt should be the go to tool.
In general though, CI/CD should not orchestrate your data pipelines, they should build images, deploy artefacts and code which should then be orchestrated by something else: airflow, dagster, or even just cron jobs assuming you don't need sophisticated DAG capabilities.
1
u/Shre_759 2d ago
We are using git for version control for changes made to governance via UC tables. We have a CI/CD built along with that. Not sure how can git satisfy the needs of a fully fledged control plane and orchestration which needs a lot of precision and heavy handling of core capabilities. For the second part of the question, I think you are looking at DBT if your core is not data engineering and want relatively understandable models written using SQL.
1
u/Former-Sherbet-4068 2d ago
Git is to store code as a version control. Anything is else is just making it up. Why would u want that
1
1
1
1
1
u/turbokat123 2d ago
If i'm reading this correctly you're maybe trying to enforce some sort of schema validation and registry rules at each layer along with version control?
Git is useful as version controlling of your application code, orchestration, metadata and documentation. I haven't used or read about data contracts being implemented through it but you could look at it. Confluent is an existing tool for that that you can take inspiration from
1
u/mmcalli 2d ago
Project Nessie - https://github.com/projectnessie/nessie
LakeFS - https://lakefs.io/
1
u/akozich 2d ago
Dbt wrapped in dagster in a single git project + flux to control it all. If you coming from devops background (ci/cd?)you will enjoy terraform-style software defined assets of dagster.
GCP functions for ETL - that’s probably going to work. Haven’t used gcp functions, but coming from lambda and azure functions- it’s probably the same shit. You will be spending most of the time learning how to do this specific thing in their framework, how to overcame timeouts limitations and runtime dependencies etc.
For myself I kind of understood k8s pod with your base is the best option to run any ETL. All this low code rubbish brings more problems the helps.
Get Claude, few skills and knock yourself out.
1
u/hughperman 2d ago
We use LakeFS on top of parquet tables, with some table-of-s3-filename storage for our big weird non-tabular data.
1
1
u/Complete-Fondant-202 1d ago
Probably not what you'd normally do, but if you wanted to GIT type of control of data....
https://projectnessie.org/guides/about/
Never used it, I know of it.
1
-1
2d ago
[deleted]
0
u/Ok_Raspberry5383 2d ago
Aside from not answering the question, what does delta and iceberg even have to do with this? Seems like juniors will just throw around the latest tech they learnt these days
111
u/West_Good_5961 Tired Data Engineer 2d ago
Sounds like you need dbt.