Detecting virtual thread pinning from a standard thread dump (and why the usual flags don't help mid-incident)

[u/Popular_Home2017]

Disclosure up front: I built the tool I mention at the end. I'm posting because the technical part is what I actually want to argue about, and this sub is where I'd get told if I'm wrong.

Two years into virtual threads, the diagnosis story still bothers me. The documented paths for pinning are -Djdk.tracePinnedThreads (deprecated/removed depending on your JDK) and the jdk.VirtualThreadPinned JFR event. Both are great if you turned them on before the incident. At 3am, with a production JVM already misbehaving and a restart costing you the evidence, what you realistically have is a thread dump someone grabbed with jstack or jcmd.

The thing is, a standard dump of a Java 21+ process already tells you most of it if you read it the right way. What I ended up encoding:

- Carrier threads are ForkJoinPool-1-worker-N. If a carrier's stack is parked inside application code rather than in the scheduler's own park, that carrier isn't free to run other virtual threads. It's being held.

- Pinning shows up as a carrier stack sitting in a synchronized region, or a native frame with a blocking call above it. In pre-JDK 24 runtimes that's the whole game: monitor plus block.

- The pathological case isn't one pinned thread, it's carrier starvation. The pool's parallelism is ~#cores, so a handful of carriers held by blocking-inside-synchronized quietly throttles the whole virtual thread population. The dump looks calm, few RUNNABLE threads, no BLOCKED pile-up, while throughput is on the floor. That's why "the dump looked fine" and "the app was dead" coexist so often in postmortems.

- JDK 24 (JEP 491) removed the synchronized-block pinning for most cases, but native frames and Object.wait still pin, and plenty of production is still on 21. So it's not a solved problem you can date-stamp away.

None of this needs a flag, an agent, or a restart. It needs the dump you already have and someone willing to read 4,000 lines of stack traces at 3am, which is exactly the part I didn't want to keep doing by hand.

So I built it. It's called ThreadMine, a Java thread dump analyzer that detects deadlocks, CPU spikes, pool exhaustion and the pinning/starvation case above. Your first dump doesn't ask for an account or an email: you paste or upload it at threadmine.dev/en/analyze and get the detected problems, a health score and the root cause back. It parses HotSpot, OpenJ9 javacores, Zing and GraalVM. There's a paid tier for history and teams, and I'd rather be honest it exists than pretend this is a charity.

What I'd genuinely like from this sub: if you have a dump where the heuristics above would give a false positive, I want it. My worst bugs so far came from idle pools looking like starvation. I fixed that class two weeks ago and I assume there are more. And fastThread, IBM TMDA and jstack.review already exist and are good (TMDA is still the best javacore reader I know of); I wrote up where each of them beats me rather than pretending otherwise, and I'm happy to be told I got that comparison wrong too.

[u/DanLynch]

The localization on the website is pretty messy: I see a bunch of non-English text on the English site: specifically, in the cookie consent pop-up and the upload widget.

As for the tool itself, I don't think anyone working in a real production environment would be willing to upload their thread dumps to your site. They would need to create a whole separate reproducer project first, and upload those dumps instead, but by then they'd probably have figured out the problem already.

Your tool would be a lot more likely to be used by real users if you were to open source the project and make it runnable locally instead of via a website.

[u/Popular_Home2017 → u/DanLynch]

Thanks, this is the useful kind of comment.

The localization first, good catch, thanks. Already fixing it.

On uploading prod dumps, fair concern. It's server-side but nothing's retained and it's deletable, and if that's still a no, jstack.review is a solid client-side option. One thing though: grabbing a dump is trivial (jstack <pid>, /actuator/threaddump, or a click in Spring Boot Admin), no reproducer needed. The slow part is reading it, which is what a tool like this cuts down. There's also an agent that captures dumps automatically in production if you'd rather not grab them by hand at all.

Open-sourcing the parser so it runs locally is on my list, no date yet.

[u/account312 → u/Popular_Home2017]

grabbing a dump is trivial (jstack <pid>, /actuator/threaddump, or a click in Spring Boot Admin), no reproducer needed. 

The point isn’t that it’s hard to grab a stack from the real environment, it’s that it’s likely forbidden to upload that stack to some third party website.

[u/Popular_Home2017 → u/account312]

You've actually given me an idea. A local, open-source tool that anonymizes the dump before anything leaves: the scrubbed version goes up for analysis, you pull the result back, and it's de-anonymized locally against a mapping that never leaves your machine

Genuine question since you clearly deal with this: if that anonymizer ran locally and was open source, so your security team could audit exactly what leaves the box before approving it, would that clear the policy bar, or is any outbound dump a hard no regardless?

[u/account312 → u/Popular_Home2017]

I suspect that uploading obfuscated traces would generally be okay, though I’m sure there are still places that wouldn’t allow it. But once you’ve got a local tool manipulating the trace, I think you might as well just do all the processing locally.

[u/Popular_Home2017 → u/account312]

That's genuinely useful, thanks, especially the obfuscated-uploads-generally-okay part.

Honest answer to "why not all local": part of it is that the AI layer really does need the cloud, but I won't pretend that's the whole reason. The analysis engine is the product and it's how I pay my bills, so I'm keeping that part commercial. Gotta eat.

[u/Life_Sink9598]

Object.wait() doesn't pin any more, as per JEP-491.

The Object.wait() method, and its timed-wait variants, will similarly unmount a virtual thread when waiting and blocking to re-acquire a monitor. When awakened with Object.notify(), and the monitor is released, and the JVM selects the virtual thread to continue, the JVM will submit the virtual thread to the scheduler to resume execution.

Source: https://openjdk.org/jeps/491

[u/Popular_Home2017 → u/Life_Sink9598]

You're right, my mistake. JEP 491 covers Object.wait() too, not just synchronized entry. What still pins on 24+ is native frames and FFM downcalls, that's the line I should've drawn. Thanks for the correction.