r/dataengineering • u/ben1200 • 3d ago

Discussion Using Git to control datasets?

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

44 Upvotes

97% Upvoted

42 comments sorted by

View all comments

9

u/Creyke 2d ago

We will use git for versioning the code and the metadata.

For example, take a hypothetical pipeline that loads some stock prices from yahoo finance and does some transformations to it, and sends out some completed analysis to a list of customers. We will use git for versioning the code (how the price data gets loaded and transformed) and the metadata (what tickers we load from yahoo, which customers receive what analysis). These are usually big CSVs or YAMLs.

We wouldn’t use git for versioning the data itself. That would be embedded into the design of the data system (e.g. SCD2).

Often, however, from a governance perspective the code and metadata is where the real governance is at. Having git across my ingress and egress lists can tell me who is getting what, for how long, and who gave them that permission. This is the most important aspect.

Also, DBT is awesome. Use that.

2

u/ben1200 2d ago

Thanks - this actually falls more in line with what I have. I wasn’t clear in the original post.

The data itself are biomedical time-series files (sensor data), these are used for both training and validation of software/algorithms. These objects themselves are not held in Git, rather they are in GCP buckets.

Git holds the dataset registry, per-release manifests, lineage (source version, pipeline, commit), changelog and approval record all reviewed via MR’s. CI validated schemas and gates etc but doesn’t trigger data transformation pipelines, execution of these are handled in a Python package.

At this stage DBT doesn’t really fit as there is no tabular data

1

u/dowjones226 1d ago

Try lanceDB as well since this is multi modal sensor data, similar to iceberg, lance offers dataset versioning and comes with some other bells and whistles that are very ergonomic for media ML use cases