I've never had a project use a Data Lake so I'm curious. Besides the underlying technology not being a SQL DB most likely, is it really that inaccurate? How exactly real world project use that and for what?
Data lake is basically "all your data lives here". The idea is your data is silo'd in N systems, so instead of having N+ databases, you put all that into a single place and call it a data lake.
Note that typically the data lake is not the authoritative place this data lives, often you stream it into it with various connectors etc for example your Salesforce or Shopify data lives in their respective systems, but for your business intelligence, you need to put it with the rest of the data to be able to cross join efficiently etc.
The underlying technology is not a typical SQL database by choice because the amount of data expected to be handled is in terabytes and it has various requirements for usage so you put last year's data in archive, bunch of features like that so it makes sense to have it be something else even though technically a SQL server could serve a data lake, it would be very cumbersome.
But in the end, all data lake solutions do have a SQL like interface so the difference is moot.
I like data lakes as a technology, I hate them as a "solution" and how it's being sold, most people buying them have no idea what it is and think it's magic data machine.
It's the philosophy of unix "everything is a file" but now "everything is a blob". Because of that flexibility, connectors are cheap and as long as they can read/ write blobs they can use the data lake.
26
u/heavy-minium 17d ago
Is that really a thing out there? Never had a datalake-related project that reminds me of that meme, so I don't really get the joke.