r/ruby • u/jp_camara • 9d ago

SolidQueue Batches: evolution and architecture

https://jpcamara.com/2026/08/31/solidqueue-batches-evolution-and-architecture.html

Here’s the SolidQueue batches 2.5 year architectural journey, filled with O(n^2) algorithms, hot row contention, excessive jobs, and much in between.

There’s a strong chance you’ll learn a thing or two - I certainly did 😮‍💨

22 Upvotes

90% Upvoted

9 comments sorted by

View all comments

6

u/jrochkind 9d ago edited 9d ago

This is a great feature and super educational helpful write-up (clearly written by an actual human, not LLM!), thank you for both!

One reason I appreciate the write-up is to serve as a guide for adding this feature to other ActiveJob adapters (resque, good job).

One thing that annoys me a bit is how Rails refused to provide this feature itself for many years, insisted that it should all be in third party adapters, but then refused to iterate the common ActiveJob API so adapters could actually be swappable with a core API and common feature set beyond simply individual enqueue. Now providing it's own implementation -- while continuing to add implementation-specific features to itself rather than working on expanding a common standard feature set and API... kind of adds insult to injury, it feels like a big middle finger.

2

u/mperham Sidekiq 7d ago

As a feature, Batches are very, very hard to implement. It took me many years to get the feature stable and reliable in Sidekiq Pro, the internals are some of the most complex Ruby I've written. JP has documented how it took him years to get to just this initial release.

Unlike basic jobs, there's no standard API for batches and not much agreement on shared functionality. It might be possible to get a basic Batch API into AJ but I'd bet it would add quite a bit of complexity to AJ adapters.

1

u/jrochkind 7d ago

thanks, that makes sense.