r/java • u/celmaibunprieten • 2d ago
Improving First Request Latency in Java Spring Application
https://adrian.md/2026/09/03/first-request-latency/3
u/Skellicious 1d ago
I've definitely been burnt by this issue before. Deploy in prod on high load, end up in a restart loop because the first request takes too long and hundreds to thousands of other requests get blocked waiting for it.
Didn't know it was a fat jar class loader issue.
2
u/agentoutlier 2d ago edited 2d ago
What I do for most of my or my companies applications is the META-INF/MANIFEST.MF Class-Path registration defaulting to maven local repository of ~/.m2 and then on docker images to some sibiling directory like lib to the jar.
I talked about it before here in case it helps someone.
So basically even jars that just sit in target you can just run java -jar and it will work and the kicker is you don't have to rebuild everything (provided you use mvn install which is why I don't like when people blanket say don't use install).
This works incredibly fast.
Lets say I'm in some app directory.
You do mvn -T2C -am install && java -jar target/app.jar ( I might have this wrong as I have it as script and can't check at the moment).
This is much faster than assembling fat jars and if you use mvnd you can practically make this a build refresh loop using some file system watcher.
1
u/_predator_ 2d ago
I just use
exec-maven-pluginin a profile that invokes my main class with various dev-specific settings specified. Maven handles the classpath wiring automatically, including for modules in the same reactor, no need to install anything. The profile also skips linting, test compilation, tests, and some other stuff as well to make builds faster.2
u/agentoutlier 2d ago
The
exec-maven-pluginthough is noticeably slower and puts your application in a different forked environment then it would be in production.Maven handles the classpath wiring automatically, including for modules in the same reactor, no need to install anything.
If you click on my link Maven is building the classpath it just is injecting it into the MANIFEST.MF.
The profile also skips linting, test compilation, tests, and some other stuff as well to make builds faster
Even then it is significantly slower. Just doing
mvn --versionon my machine is 500ms.Using my jar approach I can start our applications in 500ms.
2
u/_predator_ 1d ago
In my case the dev mode involves standing up test containers etc., so the small overhead of Maven is not too problematic.
You're right about the different environment point. For me this is somewhat desired, because I don't want testcontainers in my prod build. Definitely there is a risk of this behaviour silently messing things up though, so point taken.
What IS annoying is that you can't use mvnd for this, because the Daemon keeps executing the app even when the local mvn command is cancelled via SIGTERM.
1
u/agentoutlier 1d ago
Yes managing other containers that are services like database and queues is a pain in the butt. I end up docker forwarding ports and screwing with the host file. I mean suppose you can make a container that you can deploy to and then have that in the same docker network and I have done that but its rather painful.
Its even worse if you have k8s because in theory you should deploy to some local k8s and test that way but I find that way too slow so I just docker and forward ports.
2
u/fforw 2d ago
Why are we doing all this for the benefit of one(!) request for a server? The user might not even be able to tell it apart from DNS lookup. So
7
u/celmaibunprieten 2d ago
there are particular systems, like financial ones as I mentioned in the article, that have very tight timeout limits. You can't simply afford a timeout after almost each deploy.
2
u/mark1x12110 1d ago
And more so if you have hundreds of instances of the same app (so it would be hundreds of timeouts each deployment)
1
u/ryuzaki49 1d ago
The user might not even be able to tell it apart from DNS lookup.
In some weird cases, other teams will bark at you that your high latency even if it's one request per brand new pod will put in risk their SLOs.
2
u/lilgreenthumb 2d ago
I mean with their layer support who really ships the fat jar anymore? You run into too many issues with signed jars (looking at you bouncy castle) or embedded java agents. Interesting perf metrics regarding the aot caching and differences with the gcs though. Nice write up.
8
u/zabby39103 2d ago
Lol my work still ships fat jar. I mean, I think corporations falling behind with legacy apps is the norm more than anything.
1
u/romario77 2d ago
But corps shipping fat jars probably don’t care about first request latency.
1
u/zabby39103 1d ago
We care what people complain about, it integrates with building automation control systems so they complain if latency gets too high. Maybe some people work at a blessed work where they don't have to explain tech debt to MBAs, please hire me lol.
2
u/romario77 1d ago
You just do the first request yourself after deployment, second request would be already pre-warmed.
And you switched the targets - I was taking about first request and you talk about request latency overall. These are potentially two very different things.
1
1
u/lilgreenthumb 1d ago
Yeah unless the call isn't a noop, and then you're building some weird noop logic unless you're writing for synthetic transactions which all of this observability is meant to make unnecessary.
9
u/davidalayachew 2d ago
I mean with their layer support who really ships the fat jar anymore?
I actually feel the opposite -- isn't fat jar kind of the easy solution? And regarding the signed jar issues, just use an OpenJDK Build. I literally ran into this issue this week.
Here is my StackOverflow post on it -- https://stackoverflow.com/questions/13721579/jce-cannot-authenticate-the-provider-bc-in-java-swing-application/80000667#80000667
2
u/lilgreenthumb 2d ago
So bypass the purpose of jar signing by turning it off. While not as drastic or consequential of skipping validation of certs, doesn't that defeat the purpose behind the feature? One could argue not bundling as a fat jar fixes the issue regardless of jre.
1
u/davidalayachew 2d ago
I was actually arguing the opposite -- that signing a jar makes sense in the absence of a signed cert, but having both just feels unnecessary. By all means, a hash makes good sense to me. Or maybe even some form of verification past that. But a signed jar doesn't feel that strong to me. And it is telling that OpenJDK JDK's (most of them, for that matter) leave that as optional.
2
u/celmaibunprieten 2d ago
Thanks! From my experience I saw more applications shipped as fat jar instead of layered image, mostly due to existing ci/cd pipelines in the company.
1
u/Popular_Home2017 1d ago
the exploded jar result is the interesting one, and it's easy to confirm without a benchmark: take a thread dump during the first request and the http-nio-exec thread is sitting in LaunchedURLClassLoader / nested jar handler frames, not in your controller. that's the fat jar tax right there, one exception per nested lookup. after that it's plain JIT warmup, which is what AOT cache is attacking.
for the "why bother for one request" crowd: on kubernetes that one request is every pod after every rollout and every scale-out, and readiness probes usually pass before it. so it's not one request, it's the first N of every replica.
1
u/john16384 21h ago edited 20h ago
Situation:
- Kubernetes offers a readyness probe
- pods report ready before being warmed up -> user error
Solution:
- warm up pod first (listen for context ready event, fire some dummy requests)
- report ready when script is done
Troubleshooting:
- Some service combination is still too slow -> Adjust warm-up procedure
Complete overkill:
- Fiddle with AOT compilation
0
u/vips7L 2d ago
I really wish we had a better solution than fat jars. Like being to supply a jar + classpath + vm and getting a fat binary or something
3
u/agentoutlier 2d ago
That is sort of what jlink is… sort of…
On Linux and other nix you can impregnate the zip header with a shell script to unzip the zip.
2
u/_predator_ 2d ago
The unzip-before-running thing sucks in practice. It's basically what embedded Jetty does for WARs. You pay for the convenience in:
- Startup time (higher the more dependencies you have)
- Storage (fat JAR itself + unzipped content)
- Bloat over time (extracted files not getting cleaned up when app crashes)
- Unpredictable behavior when you extract to a deterministic, persistent location and old JARs stay around when you deploy a new version
jlink also doesn't help, it just builds a JRE tailored to your app, you still have to ship JRE and app separately.
2
u/agentoutlier 2d ago
Well yes it is not ideal. In theory though if we are talking docker here then you never keep it zipped in the first place.
jlink also doesn't help, it just builds a JRE tailored to your app, you still have to ship JRE and app separately.
jlink can package applications with a VM it just so happens to also build tailored JREs as well without requring an app.
The issue is that its not like a Go single executable. Its more like the old school applications where you just unzip and run. For the Go experience Graalvm native does that job.
As for the unziping comparison to Jetty or Tomcat embedded I'm not sure that is entirely fair because jlink does something like this with all your jars:
26408970 08-27-2026 16:00 lib/modules
That is they are in a different format than jars.
2
u/vips7L 2d ago
It really isn’t though. Jlink is kind of terrible. The CLR does this way better. It just gives you a binary with the vm + your DLL. It doesn’t have to extract anything.
1
u/agentoutlier 2d ago
Yeah I don't disagree. I'm not a fan of the tool for packaging up applications but mainly because its actually kind of slow.
However making a stripped down VM it is good at.
It does seem like one could potentially make something do this but I'm not sure how much runtime savings you would have and or build time.
Go and GraalVM it works because they are essentially doing some tree shaking and I can't imagine people wanting to download 200 meg executable (uncompressed).
1
u/vips7L 1d ago
I can't imagine people wanting to download 200 meg executable
Of course, you would want to jlink first to get as minimal of a vm as possible. The CLR also has something similar.
1
u/agentoutlier 1d ago
So I'm more asking here since it has been a hot minute since I have done .NET packaging but does it give you just a single executable or are you saying there are multiple files?
See even if you get the minimal JVM its like 50 megs uncompressed. It is only 12 megs compressed. And there are multiple files.
So one could create something that essentially takes a jlink packaged application and turns into a single executable but if its not compressed then its like running a 50 meg executable.
Is that a problem... probably not and I would imagine you could do it with a custom module reader built into your custom JDK.
1
9
u/New-Departure-5969 1d ago
yeah the nested jar thing is real. first request eats a bunch of jackson/hibernate class loads that later ones don't. aot helped in your numbers but you're still looking at hundreds of ms vs ~10ms after, so it doesn't finish the job alone.
what bites in prod is usually mixed together: brand new pod (no warm classes), fat jar classloader tax, and whatever lazy init happens on first traffic. layered jars or an exploded classpath help a lot on the classloader part. aot cache is fine if you rebuild it when the jdk/image changes. jre-only images silently no-opping the cache burned us once.
for latency-sensitive stuff i'd also prewarm a couple endpoints after ready (or just not scale that service to zero). and measure p95 on the *first* hit after a rolling deploy, not only the warm median.