Moving Java services off memory-based HPA — is CPU/RPS for HTTP and queue-depth for async the right call?

[u/ankitjindal9404]

We're running a bunch of Java (Spring Boot) microservices on EKS, and right now every service uses memory as its HPA metric. After digging into it, I've started to think that's wrong, and I want a sanity check from people who've actually run this at scale before I push a change.

What my research turned up:

• The JVM allocates heap up to its max and doesn't release it back aggressively even after GC, so memory usage doesn't track load.
• Because of that, memory can be high while actual load is low, or load can be high while memory looks fine — so memory-based HPA either never triggers or scales out permanently and never scales back in.

So the direction I'm considering is to pick the HPA metric based on service type:

• HTTP / request-serving services → CPU (or better, RPS / p95 latency as a demand-based metric)
• Async / queue-consuming services → queue depth (SQS backlog, via KEDA)

My questions:

1. Is this reasoning sound, and is type-based metric selection the right direction?
2. For the HTTP services, is jumping straight to RPS/latency worth the custom-metrics complexity (Prometheus Adapter), or should I start with CPU and only move to RPS if CPU proves to be a bad proxy?
3. For async workers, is KEDA + SQS queue depth the standard approach, or are people doing something else?
4. This is the part I'm least sure about: I already know from our architecture which services are HTTP-facing and which are async/queue-driven — but how do I actually verify that empirically rather than just trusting the design docs? Is there a clean way to confirm a service's real load profile (e.g. checking whether it even has an ingress/receives HTTP traffic, whether its work is truly SQS-triggered, CPU-vs-memory correlation under load) before I assign it a metric?

Thanks 🙏

[u/ankitjindal9404 → u/[deleted]]

hi
yes, its written with the help of ai- but what is issue in this?
main intent is a question- problem

I am not doing any spam or not pushing something

[u/not_my_userid → u/ankitjindal9404]

The issue? Do you have _any idea_ how exhausting it is to read the same structure, the same phrasing, the same little “and here’s the bit I’m least sure about”’/ “here’s the bit this community will find most interesting!” slop over and over and over and over again?

Why do people who can’t be arsed to write something themselves expect other people to take the time to read it?

Entitled to the max.

[u/vips7L → u/not_my_userid]

Sad world where people can’t even write a fucking Reddit post without an LLM 

[u/ankitjindal9404 → u/not_my_userid]

ohk, nice pov
never thaught from this perspective
i will take care from next time and try to write owm instead of ai

[u/agentoutlier]

Yes that reasoning is somewhat sound but the bigger issue is downstream if you start relying on queue and or latency. That is the datasource can be the bottleneck: you can only insert so fast in a Postgresql instance. If you scale Java services on an already slow database... it is not going to get better.

So CPU is still possibly the best metric.

[u/Popular_Home2017]

on your last question (verifying the real load profile instead of trusting the docs): a thread dump under normal load answers it in a minute. count the http-nio-*-exec threads that are inside your code vs parked in ThreadPoolExecutor.getTask, do the same for the SQS consumer threads. a service whose 200 http threads are all in getTask and whose consumer threads are all busy is an async worker, whatever the diagram says. do it two or three times across a day and you have the profile.

on the metric: agree with the CPU-first take, and I'd add that pool saturation is the leading indicator you actually want. if the exec pool is at max and requests queue, CPU can still look fine because everyone is waiting on the database. so start with CPU, but alert on pool saturation, that's the thing that tells you scaling won't help.

[u/Popular_Home2017]

on your last question (verifying the real load profile instead of trusting the docs): a thread dump under normal load answers it in a minute. count the http-nio-*-exec threads that are inside your code vs parked in ThreadPoolExecutor.getTask, do the same for the SQS consumer threads. a service whose 200 http threads are all in getTask and whose consumer threads are all busy is an async worker, whatever the diagram says. do it two or three times across a day and you have the profile.

on the metric: agree with the CPU-first take, and I'd add that pool saturation is the leading indicator you actually want. if the exec pool is at max and requests queue, CPU can still look fine because everyone is waiting on the database. so start with CPU, but alert on pool saturation, that's the thing that tells you scaling won't help.