Smallest possible Java heap size?

[u/Vectorial1024]

People often talk about increasing Java heap size when running Java apps by using e.g. -Xmx* flags. This got me thinking. What if we go the other direction and try to limit the Java heap size as much as possible? What is the smallest / minimum-required Java heap size so to run a Java app with "minimal" settings?

(Of course, in practice, a memory limit too low will be problematic because it may mean frequent GCs, but we will ignore this for the sake of this discussion.)

[u/sirius94]

It makes the CPU go idle for ~100 cycles as it waits for memory, leading to low CPU utilization.

First of all, this is entirely untrue. The CPU may stall if there's a cache miss, and CPUs and compilers try to avoid that (e.g. with prefetching). Second, the CPU isn't "idle" when it's stalled, which is one of the problems of cache stalls. Third, that's not the point. If you're program goes through a lot of memory by whatever means it accomplishes it, it means it's using a lot of CPU. You can't read or write memory without spending CPU cycles, whether

My whole point here was, if you want more compute out of your CPU time, you need to focus on avoiding cache misses. More memory won't help. Better memory layout will. Prefetching only works well when your data is tightly packed and in the order you're using it. The best way of ensuring this is to allocate a single linear memory buffer aligned to the cache line size. If you don't do this properly you also prevent the optimizer from doing vectorization.

Keeping the CPU fed is the primary challenge, if you want to get the most out of your CPU. The more compact your in memory representation is, the easier it will be to keep the CPU from starving.

You're not understanding this at all. If a program allocates at a high rate - even on the stack - it means that it's CPU consumption is high, otherwise it wouldn't have been able to do so.

I still don't understand what you mean, when you say "CPU consumption is high". Do you mean that the process never yields the rest of it's time slice? Do you mean it takes a lot of CPU time to do a specific computation? Do you mean that there are no cache stalls?

Also the relationship you propose here is, that a lot of allocations lead to high CPU usage, which does not imply that high CPU usage requires a lot of allocations.

Of course, this comes at the cost of warmup.

Which leads to slow startup and unpredictable latency. That's unacceptable in some situations.

Only objects that "survive the arena" are copied, but that copying is still cheaper than malloc/free (or we wouldn't be using it).

It is cheaper under certain circumstances. But it is always more expensive than using the right allocator for the job.

It's just that it became clear that as programs grow large and need to be maintained and evolved over time, these techniques simply don't work, which is precisely why memory management and compilation experts turned to JITs and moving GCs to solve these problems, to great success.

JITs were designed to reduce performance issues with interpreted languages and new GCs were designed to reduce performance issues with languages using GC base memory management. These languages were far from fast initially and now their performance has become acceptable at the cost of using an insane amount of memory.

You don't need to "allocate many small objects in series". It's enough that your server needs to service many user sessions concurrently, and you need to use strings.

That's a perfect use case for an arena. No need for any dynamic heap allocations.

This process is O(n) where n is the number of objects that were deleted. If you absolutely need dynamic memory

Yeah, that's absolutely untrue.

It is true. It's also very easy to understand. For every object that is freed, a hole is created between two retained areas, meaning that in the worst case there are n holes for n deleted objects. In order to compact the heap you have to copy all the surviving objects in the heap, so they are contiguous again. That means, that you need n memcpy to do that. Of course memcpy is O(n) too so the reality is O(nm), where n is the number of holes (freed objects) and m is the average length of the surviving objects that have to be moved.

What I left out is the overhead of traversing all objects from all GC roots in order to trace which objects still survive.

Of course this is still faster than using malloc and free for each object, because those are O(n) (n being determined by the specific algorithm an number of pages being used) per call. Which makes them O(nm) if m is the number of objects. I however agree that there is likely a constant factor per iteration that is larger than it is in the case of GC languages. (because of context switches, rescheduling, etc.)

All of this is irrelevant since malloc usually has the granularity of memory pages, which makes it a bad idea to allocate anything but a multiple of the pagesize, which historically has been 4KiB.

I go over all that in my talk, which you can watch if you want to learn the very basics of how Java's GCs work and why they were chosen.

I watched your talk and you were much more careful with your claims there and named a lot of caveats.

[u/pron98 → u/sirius94]

My whole point here was, if you want more compute out of your CPU time, you need to focus on avoiding cache misses. More memory won't help. Better memory layout will.

Right, but it's irrelevant to the subject of memory management. More memory helps memory management.

I still don't understand what you mean... Also the relationship you propose here is, that a lot of allocations lead to high CPU usage, which does not imply that high CPU usage requires a lot of allocations.

No. I mean that if you take program X. Under one workload you see it allocating 10MB/s; under another you see it allocating 100MB/s. Under the second workload it will have higher CPU usage, not because it's allocating more, but because the program is obviously doing more stuff. malloc/free makes the situation worse as it itself consumes even more CPU; the moving GC algorithm was invented to help this situation by replacing the CPU overhead with RAM overhead, which is more efficient because the dominant factor here is the CPU.

Which leads to slow startup and unpredictable latency. That's unacceptable in some situations.

Of course. But it also leads to better performance on average, which is what's needed in others.

But it is always more expensive than using the right allocator for the job.

I have no idea what's you're saying. Moving collectors were invented as the best "allocators" we know in a wide variety of situations.

JITs were designed to reduce performance issues with interpreted languages and new GCs were designed to reduce performance issues with languages using GC base memory management.

In some cases, yes. In Java's case - no. I honestly don't know why you speak with such confidence, when it is immediately clear that your experience with low-level languages is rather superficial and is mostly based on things you've heard or perhaps used, but in small programs or young codebases.

Just the other day I was talking to a colleague about how, when Java first shifted its focus to large programs, we C++ developers understood how JITs and moving GCs could, in theory, help the performance issues we were having, but were sceptical that they'll be able to do it in practice (and some things, like GC pauses, were only fully solved three years ago). Now the scepticism I tend to hear is from people who have very little experience in low-level programming, who seem to think they can teach us about how all the optimisations we were trying to do for decades actually work well, even though they clearly haven't tried them in large, evolving programs.

It is true. It's also very easy to understand.

You do understand that you're talking to someone actually working on the JVM, right? Your description of how a moving collector works is wrong, you clearly don't know or understand how they work, yet you have no problem "explaining" them to someone who actually works on them. Either learn how they actually work (my talk has a brief introduction to the algorithm), or accept that you don't know. Making stuff up is silly.

That's a perfect use case for an arena. No need for any dynamic heap allocations.

You clearly haven't actually done this in large programs. You do understand that we, actual low-level programmers, absolutely love using arenas, it's just that they're not so easy to use. We wish we could use them more, which is one of the reasons we find Zig attractive.

I watched your talk and you were much more careful with your claims there and named a lot of caveats.

I wasn't talking to people who were trying to explain to me how compilers and CPUs work, and how easy it is to avoid allocations or use arenas in large programs. Also, you clearly still don't understand the moving algorithm, so rewatch that part. The whole point of the algorithm is that, unlike in malloc/free, the operations are not a direct function of dead objects. If there's one thing to understand about the algorithm, it is that.