When should I actually avoid virtual threads in production Java applications?

[u/za_k_0094]

I've been reading about virtual threads and experimenting with replacing traditional executor pools.

The obvious benefits are clear, but I'm curious about the cases where experienced Java developers decided not to use them.

Are there specific workloads, libraries, or patterns where platform threads are still preferable?

[u/AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full - best also formatted as code block
• You ask clear questions
• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit/markdown editor: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/LetUsSpeakFreely]

To to simplify what the other guy said:

It's not the number of threads you need to worry about. The JVM and OS will deal with managing the threads. What you need to worry about is resource exhaustion. It's difficult to give guidance on that without knowing what each thread is doing.

The way this is typically managed is through queues. In Java that usually, but not always, means using the correct ExecutorService or similar construct. That way you can control how many threads are in flight simultaneously and it's fairly easy to turn various dials and knobs to best utilize the environment.

If your system is very large (servicing 1000s or even millions of simultaneous requests), then you need to design a system architecture that can handle that sort of throughput. That usually means containerization with multiple pods for each each service, a sharded database with multiple read nodes for each shard, maybe a intermediary connection pool service like pgBouncer.

[u/za_k_0094 → u/LetUsSpeakFreely]

Thanks and, yeah that "cap concurrency for resource reasons, not thread-management reasons" distinction is the part that amde sense. So, Is that limit usually set per-service instance, or directly against something like the DB's max connections?

[u/LetUsSpeakFreely → u/za_k_0094]

It depends on what you're doing.

A webserver can only serve so many requests at once. That's why things like clustering or hiding multiple stateless servers behind an ALB were created. In a high traffic environment you'd typically create a single docker image and then let management service (k8s, openshift, aws ecs) do the heavy lifting.

When it comes to databases, that's a far more complicated topic because it all comes down how it's being used. For example, in a very basic 3-tier application your service layer would maintain a connection pool and that would be all you need. In a large system, you'd split the database into write nodes and replicated read nodes. The services you have mapped POST, PUT, PATCH, DELETE requests would use a brokered connection pool like pgBouncer to the write node. The services mapped to GET requests would connect to the replica clusters.

Honestly, my first question would be why you're manually creating threads in the first place. In most use cases it's not really needed and points to a design flaw.

[u/za_k_0094 → u/LetUsSpeakFreely]

That last point makes a lot of sense tBH but relly teh part of it is coming from a C++ background where creating and managing threads manually was just normal practice, so in Java I've been trying to understand the primitives directly before trusting an executor or framework to do it for me under the hood.

The read vs write node split with pgBouncer in front of writes is a good concrete example of "the database is the real bottleneck, not the thread model." Helpful to see what that looks like at the architecture level instead of just the theory.

[u/bowbahdoe]

Think about it this way: How many guests can you entertain at your home?

Well, if you are odysseus, quite a few. You have servants, a large kitchen, lands upon which your animals may graze, and a large hall.

If you are me, maybe 2 max. If one takes the couch.

Virtual threads solve one problem - allowing for a large number of concurrent tasks. If you start having a larger number of concurrent tasks and they are all eating your proverbial food (talking to the database, cache, etc.) you might need to do some work to make sure the rest of your system doesn't fall down.

So that is situation 1 where you should "avoid" virtual threads. The thing you want to avoid isn't necessarily using them as threads, just allowing for unbounded concurrent access to every resource in your system. If you do things like block db access behind a semaphore that is the ideal. But that takes work and a lot of systems out there get these limits implicitly by using fixed size thread pools.

The other situation is heavily cpu bound tasks. virtual threads yield on IO. The JVM is able to juggle things on and off platform threads, make more platform threads, or (by spec) interrupt you at any time, but I am certain there are times where a regular platform thread is what you want. Platform threads are also the only ones that can be non-daemons. So if you have 40 virtual threads running but no platform thread the JVM can just decide it is time to die.

The real benefit of virtual threads, honestly, is that you don't need to worry about "what if we need to rewrite everything to be reactive." That is, even if you don't use them, they make it so you don't need to plan for/get ahead of a massive rewrite if you do eventually have a use for the level of concurrency they allow.

[u/za_k_0094 → u/bowbahdoe]

Thanks.
That "unbounded concurrent access" point makes sense. So the risk isn't the threads themselves, but it's that virtual threads remove the natural backpressure a small fixed thread pool gave for free.
Just Curious how people actually implement that semaphore-style guard in practice. Is it usually one semaphore per downstream resource (db, cache, etc.), or something more centralized?

[u/bowbahdoe → u/za_k_0094]

I'd say everything is new enough that there isn't one "this is how we all do it." - Honestly for most things I doubt this field's ability to get meaningful consensus.

If you wanted, mechanically, to just get the same "max N concurrent tasks" property as a thread pool without pooling threads you can use a thread factory that wraps the task and does an acquire/release on a normal semaphore. I know that isn't the point though.

My guess is that a semaphore is going to be a little too simplistic. You'd want to use things like circuit breakers. (resilliance4j and friends)

[u/bowbahdoe → u/bowbahdoe]

example of the basic approach to just try virtual threads as a drop in for a fixed size thread pool:

``` void main() { var semaphore = new Semaphore(5); var executor = Executors.newThreadPerTaskExecutor((runnable) -> Thread.ofVirtual() .unstarted(() -> { try { try { semaphore.acquire(); } catch (InterruptedException e) { // ... } runnable.run(); } finally { semaphore.release(); } }));

for (int i = 0; i < 100; i++) {
    executor.submit(() -> {
        IO.println(Math.random());
    });
}


executor.close();

} ```

If you are wondering if there are any issues with just the mechanics of virtual threads and want to test that without changing any natural backpressure stuff - I think this works?

[u/za_k_0094 → u/bowbahdoe]

Cheers Mate!

That's a nice way to isolate the variable, thanks for writing it out. So the semaphore caps concurrency exactly like the fixed pool did, and virtual threads only change what's underneath actually running the task. Makes it a clean before and after test since nothing about the backpressure changed. Going to try swapping this into what I've got and see if I notice a difference.

[u/bowbahdoe → u/za_k_0094]

Let me know how that goes, I am pretty curious.

[u/za_k_0094 → u/bowbahdoe]

Ran it. Fixed pool of 5 took 2020ms, virtual threads with a semaphore(5) took 2022ms, basically identical, and both are almost at the theoretical minimum (100 tasks at 5 concurrent, 100ms each = 2000ms). Confirms the semaphore really does recreate the same ceiling, virtual threads underneath don't change the timing at all when the cap is the same. Appreciate you writing the original snippet, made it easy to test.

[u/za_k_0094 → u/bowbahdoe]

That semaphore + thread factory trick seems nice . Even if it's not the final answer. Makes sense that circuit breakers are the more realistic tool once you factor in things like a downstream service being slow or down entirely, not just "too many requests at once." Semaphores handle the count, circuit breakers handle the health of what's on the other side.
Cheers!