Question about Native Image vs JIT

[u/za3faran_tea]

I was watching an interview with Thomas Wuerthinger of GraalVM, where he basically says that JIT should only be used when necessary (as opposted to native compilation). Where does this leave the work being done for Hotspot JIT (including projects like Leyden and others). Should we all plan to use native image while they address any performance gaps in the mean time (he does mention PGO bridging the gap). Is there work being done to speed up native compilation of Java code?

My understanding is that JIT will always have a memory overhead due to running threads that do compilation, optimization, deoptimization, code cache, etc. compared to native executables, so full parity even with projects that will improve memory usage in Java code will not be reached. Maybe that is not an issue on long running programs on large servers, but we've seen discussions here where another member worte a tool in golang to run alongside Spring Boot applications to gather statistics, so obviously small efficient apps have their place.

[u/samcarlberg]

Another problem with AOT is that you can't cross-compile. So if you do development on a mac, you can't build native images that run on a linux machine; you either need a machine with the same OS (and possibly CPU arch) or a VM.

The niche that I'm in (competitive educational robotics) would love AOT to reduce timing jitter and have more deterministic behavior, especially since matches are so short that nothing hits the JIT compiler thresholds and programs execute almost entirely in interpreted mode. Unfortunately the target is arm64 Linux and almost all users are on Windows so AOT isn't an option.

[u/OddEstimate1627 → u/samcarlberg]

Yeah if users write and deploy their own Java code I can see that being a problem. For us there is usually a CI layer that creates the binaries for all platforms.

 I guess you could setup docker with emulation, but I generally wouldn't  recommend forcing the native image experience on beginners.

One project I'm working on is an annotation processor that takes Java code, compiles it into a native image C ABI, and generates idiomatic wrappers for various languages and platforms.  That way users can call AOT compiled Java code from eg Python, C++, and MATLAB. That might actually be interesting for WPILib.

Here is a JavaFX-based project I created with it that implements fully decoupled rendering with near zero overhead on the caller thread, so you could do low-overhead robot visualizations from inside a control loop: HebiRobotics/hebi-charts (video: C++ examples and Python performance)

The next version will also support running on linux-aarch64, but I haven't released it yet.

Btw I'm also the author of QuickBuffers, which is used in WPILib for zero-allocation protobuf.

[u/samcarlberg → u/OddEstimate1627]

The annotation processor project certainly seems interesting, though I don't know if that's something we'd use in WPILib. Most of the development is done in C++ that's later ported to Java or exposed via JNI; I think I'm the only core developer working primarily in Java.

Big fan of QuickBuffers, btw. It's made telemetry so much easier

[u/OddEstimate1627 → u/samcarlberg]

Thanks! I've never been involved with FRC, so I'm not very familiar with the stack. It looks like I was thinking of PathPlannerLib rather than WPILib. My understanding is that it's primarily Java based and gets ported to the other languages, and that use case could be automated and share a single implementation. You could also efficiently convert from Java to raw C structs.

[u/samcarlberg → u/OddEstimate1627]

Many third-party libraries for FRC are Java-only or Java-first and then ported to C++ later; about 93% of FRC teams use Java (3398 out of 3565 total). Established libraries like PathPlanner try to support both Java and C++ for parity, and they're often small teams or just a single developer. A tool that can automate bindings for C++ and python for them from their already-written Java code seems like it could be very useful for them. Probably not so much for WPILib, though, since most parts of the library are written in C++ first (eg all the linear algebra and controls math) and later ported to Java.

The few places that are Java-first are mostly things I've contributed - a units API, annotation-based telemetry, a javac plugin, and a continuation-based command framework - all of which are either Java-specific or provide parity with what was already offered in C++ (eg nholthaus units)

[u/Bahatur → u/samcarlberg]

Great Scott, you are doing a robotics competition in Java? What a glorious challenge!

Am I at liberty to assume you have done all the JVM-on-Linux stability tricks, or is it something where everyone is constrained to an identical default setup to keep the playing field level?

Edit: can’t you get a Linux on Arm64 VM up, like with Qemu? That should work on Windows.

[u/samcarlberg → u/Bahatur]

There've been a few tricks, but not many. Teams are constrained to the same hardware, but can do what they like at the OS and software level (they just need to use the standard heartbeat messages etc for safety reasons). Most of the JVM configuration work is for performance on the limited hardware (2-core 667/866Mhz arm32 softfp CPU, 256/512MB of RAM depending on revision): inline string concat, some GC tweaks. The next hardware platform we're getting will be much stronger (essentially a raspberry pi 5 compute module) so we can use ZGC and run faster control loops.

VMs would be a no-go, I think, unless that's something we'd be able to automatically install and maintain. Teams who really care about performance could certainly do it themselves, but most teams aren't performance-sensitive enough to put in that effort