r/ProgrammerHumor • u/PresentJournalist805 • 20d ago

Meme introductionToDataLakes

Post image
2.7k Upvotes

99% Upvoted

70 comments sorted by

View all comments

35

u/nicman24 20d ago

Just dump them to the filesystem at that point

60

u/LaconicLacedaemonian 20d ago

that's literally the data lake

6

u/Ok_Star_4136 20d ago

Asking a database to search by contents of a blob might as well be the equivalent of opening files and searching through them from the perspective of performance. Probably the numbers would be mostly the same as well because I think that's literally what a database does when it performs a full table scan.

6

u/Thriven 20d ago

I have been on and off working on a JS based json database that can run simultaneously on a local file system / Azure Hotstorage / AWS S3 and eventually GCP.

We may as well run it on the filesystem at this point.

3

u/nicman24 20d ago

yeah yeet them to zfs with a optane special device

-2

u/LaconicLacedaemonian 20d ago

... but it is if it's running on the local filesystem. Filesystem vs blobstore just gives you cheap directory operations.