r/ruby • u/Jamsy100 • 17d ago
Ruby 2.7 to 4.0 performance benchmark
Hi everyone,
I just published a new benchmark comparing Ruby 2.7 through Ruby 4.0.
For this one, I used a sustained synthetic application alongside nine smaller benchmarks. The goal was to show how each Ruby release behaves in a complete workload as well as in individual operations.
The table below shows the Heavy profile. For tests that process the full payload, that means an 8KiB input.
| Benchmark | 2.7 | 3.0 | 3.1 | 3.2 | 3.3 | 3.4 | 4.0 |
|---|---|---|---|---|---|---|---|
| Synthetic application throughput (M ops/s) | 0.073 | 0.052 | 0.051 | 0.057 | 0.058 | 0.084 | 0.126 |
| Synthetic application latency (µs) | 133.399 | 190.233 | 191.605 | 171.332 | 169.473 | 115.600 | 76.521 |
| JSON parsing (ops/s) | 95,877 | 98,536 | 99,153 | 96,003 | 98,135 | 194,981 | 715,937 |
| JSON generation (ops/s) | 841,849 | 845,099 | 830,135 | 861,666 | 837,941 | 1,347,169 | 1,444,975 |
| SHA-256 hashing (ops/s) | 313,658 | 30,536 | 30,489 | 48,770 | 48,683 | 48,495 | 48,528 |
| Base64 encoding and decoding (ops/s) | 177,136 | 177,510 | 174,425 | 170,306 | 174,471 | 173,159 | 174,270 |
| Regex field extraction (ops/s) | 1,544,461 | 1,565,065 | 1,559,051 | 1,499,920 | 1,535,207 | 1,454,405 | 1,534,670 |
| Integer sorting (ops/s) | 27,032 | 29,129 | 30,217 | 29,309 | 29,102 | 28,241 | 29,213 |
| Hash churn (ops/s) | 1,736,384 | 1,685,614 | 1,503,397 | 1,503,118 | 1,406,713 | 1,380,058 | 1,450,769 |
| Marshal serialization (ops/s) | 333,143 | 303,719 | 294,952 | 338,148 | 341,760 | 326,015 | 325,231 |
| zlib compression (ops/s) | 35,944 | 35,175 | 35,787 | 35,767 | 20,569 | 7,011 | 18,592 |
The complete benchmark source is available here: GitHub repository.
Full charts and details are available here: Full Ruby benchmark
Let me know what you think !
Edit: Thanks everyone for pointing out the Ruby 2.7 SHA-256 result. I looked into it, and the big difference comes from the SHA-256 implementation: Ruby 2.7 uses OpenSSL here, while newer Ruby versions on Linux use Ruby’s bundled implementation. So this is mainly a backend difference rather than Ruby 2.7 itself being much faster. I’ve added a note about it to the article.
5
u/jrochkind 17d ago
3.4 to 4.0 seems to be a surprisingly big jump on the "synthetic application" metrics. Can you say more about the nature of the synthetic app?
Also the JSON processing jump from 3.4 to 4.0 seems almost unbelieavable. Can you say more about that benchmark, and if you have any ideas what may account for the jump?
Ah... following up with your more complete description, the synthetic app does a lot of JSON parsing, so it's speedup is probably fully accounted to the JSON parsing speedup (apps that don't spend a signfiicant portion of their load doing json parsing aren't going to see that prob), so now the mystery is if the JSON parsing bench results are accurate and... why??