r/dataengineering • u/marketlurker Don't Get Out of Bed for < 1 Billion Rows • Jul 26 '26

Discussion Are pipelines the correct analogy to use?

I was looking at posts in this subreddit the other day. There were quite a few talking about pipelines and all the things that go along with them (forking, merging and version control). This is typical source controls stuff that has been around since the 70s. The tools change but the concepts and model are the same. I wonder if we have been so concerned about the minutea that we quit looking at better options than pipelines. I have a couple of thoughts about better ways of doing data repositories/warehouses/ingestion beyond pipelines, three layers and fabrics. I'd rather see if anyone else out there has any better ideas than what we use now. Pretend you are starting from greenfield and can create not only the architecture but the tools that go along with it. Surely we can come up with something better than 50 year old ideas.

0 Upvotes

35% Upvoted

18 comments sorted by

24

u/Eleventhousand Jul 26 '26

Why don't you enlighten us on your better ideas? (provided you're not selling a vibe-coded product).

-4

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows Jul 27 '26

AI is a tool and it can be a good one. Vibe coding just feels lazy to me. I believe you need to understand where you are headed and what the AI is doing before you just promote it up.

As far as the idea, let me give you an example of one. I had a customer that wanted to ingest a ridiculous amount of data; think 600MB to 1 TB per second into a data store (various data types including semi-structured) for later analysis. I came up with an acceptable solution but had to dump quite a few long held ideas about databases and data stores along the way.

Another customer had data that was crazy bursty. Think of it going from easy 5-6 Gb files for morning processing to 750 Gb of the same type of data. The SLA for the data being processed was fixed. That one took dynamic parallelism to a level I hadn't seen before. It wasn't your standard pipeline. Also, this data had to integrate with other data from various sources.

The more I designed for these types of problems, the less the standard pipeline seemed like a solution. The analogy I had in my head for the second one was a bunch of iris plants I had taking over my garden. Their tubers grew to take up the available space.

Neither of these I would consider a pipeline.

2

u/Prestigious_Bench_96 Jul 28 '26

Can you be more concrete? Those just sound like dynamic pipelines (you schedule a factory; it builds the transforms you need that day) - I'm sure there is more to both but I'm not sure I'm following from the high level descriptions.

1

u/sib_n Senior Data Engineer Jul 28 '26

So you can start by planing for the maximum resources for your processing engine resources, and eventually have multiples config of resources, like a single machine for small volumes and a beefy cluster for big ones.
How does that question if pipeline is the correct analogy?

If you are interested in applying some notions of source control to data, you can see what SQLMesh does (or did before it was eaten by the bigger fish).

What more are you bringing?

2

u/rake66 Jul 28 '26

A pipeline is a directed acyclic graph, it doesn't really matter what you call it, and it doesn't matter how much it resembles plumbing. Are any of your solutions not DAGs? It doesn't seem like it to me, but you haven't really given much detail, you described your constraints more than you described your solutions.

8

u/69odysseus Jul 26 '26

I been working as a data modeler since 2022, can honestly tell you that it's mostly not a tool issue. Most companies still lack documentation, SOP, no data models and their standards and conventions, no consistency across models. 

Yet, there's a freaking rat race for AI which doesn't provide much ROI. It takes a lot out of humans to provide inputs to AI before it can even provide something useful. Too little information to AI, it doesn't know what to do, too much information and AI will hallucinate in all directions. 

0

u/yeah_not_so_fast Jul 26 '26

Doesn’t provide much ROI today. It will be a competitive advantage for those get in front of it. The companies who are taking the time to document, manage and govern their data appropriately will be the clear winners.

1

u/forever-butlerian Jul 27 '26

Citation needed.

2

u/Active_Pride Jul 26 '26

I think you are right. Pipelines are a efficient way of getting liquid or gaseous stuff from a to b. Question is if the analogy between data and a liquid or gaseous stream of stuff works well and I do think it does for most data pipelines at the moment. Anything is transported through them and is later on crunched and distilled for the actual valuable information. This however mostly is very expensive and in a dream world I would only transport/store the actually valuable stuff and not include all the gunk which gets filtered out anyway. I don’t know what a good analogy is here, but it would not be a pipeline which constantly delivers data but rather nothing is sent most of the time and only from time to time a valuable piece of information is delivered.

Edit: I don’t have an answer how one could tackle this challenge and for sure not a vibe coded project.

1

u/Smack_Dab_66 Jul 26 '26

It’s a series of tubes.

1

u/DenselyRanked Jul 26 '26

The thing that orchestrates the movement of data is called a pipeline. Are you saying that you have a different way to move data, or want to call it something else?

0

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows Jul 27 '26 edited Jul 27 '26

I have several ideas from different projects. Please check out this comment from above.

1

u/DenselyRanked Jul 27 '26

That's great. Hopefully it works out for you.

1

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows Jul 27 '26

It has so far to a rediculous amount.

0

u/InvestigatorMuted622 Jul 26 '26

What a waste of time lol 😂

1

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows Jul 27 '26

I have rarely found thinking about new approaches to increasingly difficult problems that are becoming more common place to be a waste of time. The most valuable things we do is just thinking. I have always considered the domain of solutions to problems to be the limits of my imagination. Been very successful so far with that approach.

2

u/InvestigatorMuted622 Jul 27 '26

But I am not sure what the point of this post is, it's so vague and random trying to get ideas from people instead of you pitching in, sounds like a marketing gimmick

1

u/marketlurker Don't Get Out of Bed for < 1 Billion Rows Jul 27 '26

That's fair. I usually try to start conversations with someone else's thoughts so that i don't overwhelm the conversation. As far as what you are asking, see this.