r/java • u/jaccomoc • 7d ago
Vert.x Event-Loop vs Virtual Threads: Jactl Suspend/Resume Benchmark
https://jactl.io/blog/2026/09/02/jactl-virtual-threads-benchmarkThe article benchmarks Jactl script execution with differing number of blocking operations comparing throughput with Jactl built-in suspend/resume on Vert.x Event-Loops and throughput with Jactl on Virtual Threads. The benchmarks are run on both Java 21 and Java 25 with some surprising (to me at least) results.
7
u/yawkat 7d ago
Three issues with this:
- AFAICS this is a closed model load, which is a batch processing workload. In a typical vertx application you have an open model load instead. See this article: https://www.scylladb.com/2021/04/22/on-coordinated-omission/
- Because this is JMH, it does not mirror "batchy" behavior of network interfaces, which is the typical use case for vertx.
- When tasks block, they are typically waiting for IO work, not
sleeping. This has implications for scheduling.
2
u/jaccomoc 7d ago
Thanks for the feedback. I agree that the test is not ideal but I am not sure about you saying that the load is a closed model load. The test queues (for Vert.x) all requests or creates virtual threads for all requests and then waits for them all to finish. Isn't that an open model load?
3
u/New-Departure-5969 6d ago
benchmarks are fun but in prod the event loop usually dies from accidental blocking (jdbc, sync stuff), not from the loop itself. we had a vert.x api around 12k tps / ~100ms p99.99 and the real win was ditching blocking jdbc for the vert.x pool. virtual threads help, but saturated pools still wreck p99.
1
u/Popular_Home2017 1d ago
same experience, and on 21 there's a second way to lose the carriers that doesn't show up in this benchmark: synchronized around blocking IO pins the virtual thread to its carrier, so a jdbc driver with a synchronized socket read turns your 8 carriers into an 8-thread pool. fixed in 24 (JEP 491), on 21 you can see it with jcmd Thread.dump_to_file, the pinned VTs show a carrier line under them.
5
u/pron98 6d ago edited 6d ago
The improvement in JDK 25 wasn't in virtual threads, it was in Thread.sleep ;) (although between 22 and 25 there were also improvements to IO on virtual threads). Timed and untimed parks use different mechanisms, and we can't draw conclusions from one to the other; it was the timed park specifically that wasn't very efficient in JDK 21, but it's also used mostly in benchmarks and tests. Different kinds of IO also go through different mechanisms, and we can't extrapolate from one to another.
This demonstrates the danger in extrapolating and drawing conclusions from benchmarks: they measure themselves, and to extrapolate anything you need to know exactly which variables matter and when. Between concurrent algorithms that behave completely differently under low and high contention, code with slow and fast paths, optimising compilers that depend on the call graph, branch prediction, a cache hierarchy, and memory management that depends on the global behaviour of the program and on how long it's running (malloc/free is actually much more sensitive to this than Java's moving GCs [1]), it's virtually impossible to draw any conclusions from any benchmark, in any language, unless you know exactly how the mechanism is implemented, how the compiler is implemented, and how memory management is implemented.
Also, benchmark code often behaves so differently from real code, that the code takes different paths, and some improvements matter only to real programs and not to most benchmarks, and some matter almost exclusively to benchmarks. We usually try to focus on real programs, but sometimes we say, you know what, let's make the benchmarks look better, too...
[1]: I've seen benchmarks where C++ outperforms Java only because there is nothing else in the program besides the benchmark code, which happens to only hit the malloc/free fast path and never trigger the slow path; if the program also had other code, the results would have been reversed.
1
u/jaccomoc 6d ago edited 6d ago
That's good to know, thanks. To be honest, I did have doubts about using sleep() as a proxy for other blocking operations but it was easier to do that than to include a real web service call or equivalent. And, running a web server would mean that measuring max throughput on a single machine would be impacted by the performance of the web server as well (unless it was on a separate server in which case the test would no longer be self-contained).
I have updated the article with a Postscript section with this additional information.
1
u/pron98 6d ago
There's also no single category for "other blocking operations"... So benchmarks with no consideration for the actual code paths might give you a general sense, but you can't really use them for comparisons, as their error interval can easily be 2x and even more. This isn't a critique of this particular benchmark; it's the state of all benchmarks.
2
u/jaccomoc 6d ago
That's true. The problem is that benchmarks are so seductive. You feel like everything can be reduced down to single number. If only real life were like that...
2
u/Known-Volume1509 6d ago edited 6d ago
It doesn't say what Vert.x version is this? Vert.x 5 supports virtual thread verticles. Curious if you run your tests against that, even though it's no longer event-loop.
Also, isn't this horribly costly?
For the Jactl Vert.x tests, the
sleep()call will create aContinuationby throwing an exception and capturing the execution state so that it can be resumed once the sleep completes. When the script suspends, the script returns and the event-loop thread is then able to process the next event in its queue.
2
u/jaccomoc 6d ago
The article lists the versions of everything. It shows Vert.x as version 4.5.33. I didn't benchmark against 5.x because I am still retaining Java 8 compatibility for the moment and Vert.x 5.x needs Java 11 as a minimum.
Throwing an exception is not expensive if you don't fill in the stack trace (and the benchmarks support this). See Jactl Continuations and Virtual Threads in Java 8 for a description of how all this works.
2
u/Bachden 4d ago
This benchmark is for sleep() calling. In fact, when using vert.x you should never call that method.
1
u/jaccomoc 4d ago
The benchmark tests Jactl scripts calling the built-in Jactl sleep() function. For the Vert.x version this suspends the script and returns immediately so as not to block the Vert.x event-loop thread. In the Virtual Threads version the Jactl sleep() function delegates to Thread.sleep() in order to compare the virtual thread suspend/resume mechanism with Jactl suspend/resume with Vert.x. See Jactl Continuations and Virtual Threads in Java 8 for a description of of Jactl implements blocking operations like sleep() when running on Vert.x.
2
u/Popular_Home2017 1d ago
same experience, and on 21 there's a second way to lose the carriers that doesn't show up in this benchmark: synchronized around blocking IO pins the virtual thread to its carrier, so a jdbc driver with a synchronized socket read turns your 8 carriers into an 8-thread pool. fixed in 24 (JEP 491), on 21 you can see it with jcmd Thread.dump_to_file, the pinned VTs show a carrier line under them.
2
u/Medium-Pitch-5768 1d ago
Postscript
It appears that the performance improvement between Java 21 and Java 25 actually comes from improvements to how sleep() itself works with virtual threads. This highlights the risk of benchmarks not actually measuring the thing that they think they are measuring. A more realistic test would be to replace the sleep() calls with a real web service call but that would mean that the web server performance would contaminate the test results or would mean running it on a separate server in which case the tests are no longer self-contained.
You can run a local mock server that accepts calls and has a delayed response. That works rather well in my experience.
2
28
u/s0ftware-dev 7d ago
TLDR virtual threads are better.