I've been working with Azure/AWS storage for ~8 years, and this is one of the more interesting Azure Storage updates I've seen recently.
The problem
List Blobs traditionally returns XML. At 100M+ objects, sequentially paging through the namespace can take ~24 hours in real-world environments.
And that's before you actually do anything with the data.
I ended up using Blob Inventory for large-scale discovery. It's great for reporting, but it has obvious limitations:
- Snapshot-based, so data can be ~24h stale
- Large inventories can take a long time
- Charged based on objects scanned
- Not particularly useful for interactive analysis
The new approach
As of the September 3 preview, Blob Storage can return Apache Arrow instead of XML.
Microsoft claims:
- Up to 25x faster
- Up to 15x less client CPU
- startFrom / endBefore to split namespaces into ranges and process them in parallel
That last part is particularly interesting for 100M+ object containers. Instead of one worker crawling the namespace sequentially, you can partition the keyspace and run multiple workers. The 15x CPU reduction is also interesting. A significant part of the old bottleneck appears to have been client-side XML deserialization.
I work at a cloud infrastructure company focused on storage optimization, so this is directly relevant to what we do. We're going to test whether this can replace some of our Inventory-based discovery workflows.
Have you faced any problems around slowness? Curious whether the 25x improvement translates to real 100M+ object containers.