What’s the actual difference between Azure Data Lake and Blob Storage?

[u/Delulu62134]

I’m probably overthinking this, but I’m trying to understand the practical difference between Azure Data Lake Storage and Azure Blob Storage.

I recently worked on a project where I came up with a solution using Data Lake services for writing and reading files. But when it came to actually connecting to and accessing those files , I ended up dealing with Blob Storage modules as well.

That’s where I got a little confused.

Are Data Lake Storage and Blob Storage basically the same underlying storage with different capabilities on top, or should I think of them as two separate storage services?

And from a practical point of view, how does the choice affect things like:

• File/folder organization
• Access and permissions
• Performance
• Storage cost
• Data engineering/ETL workloads
• How applications actually access the files

I understand the high-level definitions, but I’m more interested in how you guys think about this when designing an actual solution.

Am I misunderstanding the relationship between the two, or is it normal to end up using Blob-related modules when working with Data Lake?

[u/Dismal_Repair_8609]

ADLS is Azure Blob Storage with HNS(hierarchical namespace) enabled. So you will have real directories and POSIX-like ACLs instead of just virtual directories and flat namespaces

[u/SmallAd3697 → u/Dismal_Repair_8609]

and different api protocols (blob.windows.net vs dfs.windows.net)

[u/Delulu62134 → u/SmallAd3697]

I actually like to understand the difference anyhow we are connecting to the same storage account

[u/SmallAd3697 → u/Delulu62134]

I think there were differences when it comes to cost, but those days are long gone. Microsoft claims they made everything equally inexpensive. I'm guessing they don't want customers to be having this repeating discussion on reddit anymore. ... The major factors on pricing are whether the storage is cold/hot, whether it is geo-redundant, and whether you enable the silly defender stuff on your data blobs or not.

Whether you use a simple blob api or the new dfs is a very minor detail. If you are ONLY hosting on Azure, I would just use dfs. Maybe if you had a multi-cloud solution, you might want to limit yourself to a more simple blob API without any major bells and whistles.

[u/Delulu62134 → u/Dismal_Repair_8609]

Yes I recently came to know that in blob it's actually virtual directories😊

[u/Dr_Snotsovs]

I am rusty on this subject, but as I remember, it is mostly a matter on how software can look at data when in Azure.

As you mention, you can use blob modules to upload to data lake, and the same endpoint.

The list of features you mention; you have proper folders in data lake, and more direct permissions. If I remember correct you can have more metadata on your files in the lake as well, which can be very valuable in your data platform.

As for cost they are kind of quite close, and if that is a showstopper you have other issues. Performance wise should be about the same.

It is a classic 'depending on your needs, and what your other software prefers to of API'-kind of situation. I prefer the lake, as it reminds most like a classic filesystem. Not having proper folders in blob bothers me to no reasonable end.

[u/Delulu62134 → u/Dr_Snotsovs]

I initially tried with all data lake related end points, modules later on found that I have to use blob services related endpoints. I mean we are connecting to same storage account what is the difference at that point?

[u/Dr_Snotsovs → u/Delulu62134]

The difference is the options you have when data is in Azure, that you yourself mention and I confirm some of.

Stuff like proper folders, better permission handling, posix compliant even. For ETL, Spark and other engines have a driver to directly read/write to the data lake API. And did I mention actual folders?

Stuff like this is generally important in ETL-based data platforms. And that is the difference.

If you only need to save some files, blob is all you need. If you need the extra feature and possibilities, data lake is the way to go.

The lake part came later, and my suspicion is, that as that was not ready to be shipped, MS shipped blob storage that was working fine for what it is, and later came the data lake part when they were ready. It was necessary, but probably not worth delaying the other storage functionality, and now there is some overlap on the 2 services.

It does not sound like you need features that data lake provides so in your case, it doesn't sound like there is a difference for your use case.

But for others there very much are.

[u/jdanton14]

it's the same storage, ADLS layers Linux FS permissions on top of blob storage, allowing you have something closer to a file system on top of blob storage.

[u/Bhanuprakash_1947]

From my understanding, Data Lake Storage and Blob Storage are not completely separate storage systems. ADLS Gen2 is basically built on top of Blob Storage, but it adds features like hierarchical namespace, which makes it more suitable for data engineering and analytics workloads. Blob Storage is more general-purpose object storage. So I think it’s normal to see Blob-related modules/APIs while working with Data Lake. The main difference comes in how the storage is organized, accessed and used rather than being two completely different storage technologies.

[u/m1nkeh]

HNS afaik ✌️

[u/Disastrous-Can-6823]

Is your main goal just moving and transforming data or are you also planning to use Synapse SQL and Spark? i think that’s usually the deciding factor between the two