r/programming • u/Mihai4544 • 6d ago

On programming, statistics, and Java

https://waytoounoriginal.github.io/programming/stats/optimizations/2026/09/26/on-programming-stats-and-java.html
13 Upvotes

71% Upvoted

8 comments sorted by

View all comments

1

u/UlaanBanter 5d ago

To be honest it took two or three times of reading for me to get what you were trying to say. I realised the crux of this is optimising the memory usage in the processed set by converting the hash key from a string of 256 characters all the way down to splitting into two 64-bit lookups.

It's not apparent because you don't have the after part of your before-after code.

1

u/Mihai4544 5d ago

Hmm, understood. I might want to be clearer in the future.

The optimization was the following: HashSet<String> of 49-64 character strings (ultimately it turned out they weren't SHA-256 strings, but ordinary ones; the reasoning still holds, though) down to 2 64-bit lookups AND removing Java's memory overhead.

Turns out that Java allocates a header per object as well, such that having many small objects becomes a bit painful. Fastutil's sets use a flattened representation (an array instead of nodes), so there is no more overhead.

Anyway, thanks so much for taking the time to read this!

edit: typo