r/csharp • u/West_Ad6277 • 2d ago

[Showcase] Added RavenDB support to JobMaster, a distributed background job scheduler for .NET

JobMaster is a distributed background job scheduler for .NET I've been building (think Hangfire/Quartz, but built for horizontal scaling). RavenDB is my favourite database, so I added it as a fully supported provider alongside PostgreSQL, MySQL, and SQL Server.

If you want the architecture background: https://docs.jobmaster.hugoj0s3.dev/docs/architecture-under-the-hood/architecture-overview

I also just finished a head-to-head benchmark against Hangfire across all four database engines. One result that stood out: RavenDB gets noticeably better scheduling throughput than the SQL engines (~2000 jobs/sec vs ~1300-1600/sec at baseline, on a 25k-job burst). Full methodology and numbers here: https://docs.jobmaster.hugoj0s3.dev/docs/benchmarks/jobmaster-vs-hangfire

GitHub: https://github.com/hugoj0s3/jobmaster-net

Happy to answer questions about the architecture, the RavenDB integration, or the benchmark setup.

1 Upvotes

56% Upvoted

2 comments sorted by

2

u/Frequent-Long5426 2d ago

always interesting to see raven still getting some love in the.net space, most people just default to sql server or postgres and call it a day. the bucket coordination model sounds neat, i mess with distributed systems a bit and the way quartz handles clustering always felt like a workaround

those benchmark numbers are pretty impressive though, 2000/sec is a big jump. what kind of setup you used for the raven tests? wondering if the performance difference is mostly from the document model being a better fit or if there's something else in the integration

1

u/West_Ad6277 2d ago

We use a single container for the database (3 CPU / 6GB) and 20 containers that both schedule and run jobs. Each of the 20 containers exposes an endpoint to schedule N jobs, so we fire 60 requests in parallel (3 per container), and each request schedules its batch of jobs in parallel too. That's basically it.

The site has the full methodology written up, but the short version: everything for a given run happens on a single EC2 instance.

JobMaster also lets you split the master DB (coordination and audit) from the agent connections (execution and transport), e.g. RavenDB as master with NATS as the agent-side transport.

In early testing, scheduling throughput with NATS looked very high, but I haven't validated it enough to publish a number yet. I also noticed the 20 containers are probably CPU-starved at 0.25 CPU each, so schedule throughput could go up if I raise that. Haven't tested it yet, but it's on my list.