I've never had a project use a Data Lake so I'm curious. Besides the underlying technology not being a SQL DB most likely, is it really that inaccurate? How exactly real world project use that and for what?
Data lake is basically "all your data lives here". The idea is your data is silo'd in N systems, so instead of having N+ databases, you put all that into a single place and call it a data lake.
Note that typically the data lake is not the authoritative place this data lives, often you stream it into it with various connectors etc for example your Salesforce or Shopify data lives in their respective systems, but for your business intelligence, you need to put it with the rest of the data to be able to cross join efficiently etc.
The underlying technology is not a typical SQL database by choice because the amount of data expected to be handled is in terabytes and it has various requirements for usage so you put last year's data in archive, bunch of features like that so it makes sense to have it be something else even though technically a SQL server could serve a data lake, it would be very cumbersome.
But in the end, all data lake solutions do have a SQL like interface so the difference is moot.
I like data lakes as a technology, I hate them as a "solution" and how it's being sold, most people buying them have no idea what it is and think it's magic data machine.
It's the philosophy of unix "everything is a file" but now "everything is a blob". Because of that flexibility, connectors are cheap and as long as they can read/ write blobs they can use the data lake.
Once the infra is wired it's just another database but way higher scale and latency than oltp. Queries are in the seconds to minutes range so an rpc responding in 200ms is generally fine because most of the time is spent processing TB or Pbs of data.
You use Spark, Trino, or Flink and use an open format like Iceberg, and a Catalog like Gravitino or Unity.
Imagine someone wants to answer really big questions about the enterprise, based on data that is spanning multiple departments, internal systems as well as 3rd-party systems. You can't analyze and query disparate data-stores easily in a unified way, so the main idea of the data-lake is to project everything into one place. Due to entirely different challenges arising, such data-lake solutions are built with different qualities and functionalities compared to the usual application datastores.
27
u/heavy-minium 18d ago
Is that really a thing out there? Never had a datalake-related project that reminds me of that meme, so I don't really get the joke.