[*****UPDATE 9/21/08: I've added Rails 2.1.1 and Rails 2.2 head stats for C MRI Ruby on Mongrel. Rails 2.2 tests pretty slow and the results should not be taken seriously until the Rails team comes to a point release and optimizes. I've also added tests with the MySQL C gem and Merb seems to get more of a speed boost than Rails does.
[*****UPDATE 9/21/08: I've done some additional optimization on the JRuby Mongrel tests, and JRuby Mongrel is now approaching the speed of MRI Mongrel. On a production platform, JRuby Mongrel might be faster than MRI Mongrel since Apple's Java implementation is so slow.]
[*****UPDATE 9/19/08: the Merb team did some profiling of Merb yesterday and they fixed some slowdowns that were recently introduced. I've rerun all the Merb tests using Merb head as of 9/18/08 @ 5:36 PM and added a new column with the Merb head data. Merb is now faster in all cases: see the results below.]
I just finished running benchmarks on Merb 0.9.5 and Rails 2.0.2 across a number of servers in both MRI C Ruby 1.8.6 and JRuby 1.1.4.
The test consisted of one action that retrieves one model object via ActiveRecord and renders the contents of that object via a simple template in erb. I’ve uploaded both apps.
I did the benchmarking from a different machine using apache benchmark; specifically, I called “ab -n 1000″ repeatedly until the results converged. Apache benchmark wasn’t using any concurrency (-c command line option) so the benchmark should have been testing how fast the app runs in one core. The benchmark was run on my MacBookPro 2.4Ghz Intel Core 2 Duo w/ 4 GB of ram.
*****DISCLAIMER****: the absolute requests per second (rps) figures reported here will be significantly slower than what you would get on a production machine. The figures here are only useful when comparing speed differences between frameworks and environments.
C MRI Ruby 1.8.6 + Mongrel 1.1.4
| Environment | Rails head (rps) | Rails 2.1.1 (rps) | Rails 2.0.2 (rps) | Merb 0.9.5 (rps) | Merb head (rps) | Notes |
|---|---|---|---|---|---|---|
| -e production | - | - | 250 | 221 | 361 | |
| -e production (session :off) | - | - | 280 | n/a | n/a | Merb only establishes sessions when needed. |
| -e production (with mysql c library) | 211* | 263 | 278 | - | 440 | |
| -e production (with mysql c library and session :off) | 211* | 296 | 321 | n/a | n/a | Merb only establishes sessions when needed. |
* The results for Rails head 2.2 are preliminary and should be ignored until the Rails team releases an optimized point release.
Results: Rails is 13% faster than Merb even when it is using sessions.
Merb head is running ~58% faster than Rails 2.0.2.
Jruby 1.1.4 + Mongrel 1.1.5
| Environment | Rails (rps) | Merb (rps) | Merb head (rps) | Notes |
|---|---|---|---|---|
| -e production | 176 | 143 | 267 | |
| -e production (session :off) | 202 | n/a | n/a | |
| -e production (jruby -J-server) | 171 | 156 | 267 | To get a performance increase when using -J-server, it is important to optimize memory settings (see next line). |
| -e production (jruby -J-server, -Xmn256m -Xms1024m -Xmx1024m) | 205 | - | 296 | Memory settings used (256 MB for garbage collection, 1024MB for heap) seems to be optimal on my system. |
| -e production (jruby thread pooling) | n/a | 155 | 279 | |
| -e production (jruby -J-server and thread pool of 25) | n/a | 162 | 298 | |
| -e production (jruby -J-server, thread pool of 25, -Xmn256m -Xms1024m -Xmx1024m) | n/a | - | 334 | Memory settings used (256 MB for garbage collection, 1024MB for heap) seems to be optimal on my system. |
Results: Rails is about 9% faster than Merb running with java server mode enabled and with a thread pool.
Merb head is running 44% + faster than Rails to ~ 62% faster with all JRuby optimizations. The fastest rps here is Merb head (server mode, thread pool, memory settings) at 334 rps, which is approaching the rps of Merb in MRI Ruby of 361 rps (using ruby mysql). The Ruby MySQL adapter that ships with Rails 2.0.2 is very slow (see benchmarks above in C MRI Ruby + Mongrel) and the JRuby / Mongrel environment should be faster with a native Java database adapter (maybe the JDBC adapter?). Note also that the MacOSX implementation of java is notoriously slow (I’ve heard estimates of 30% slower than a linux java) so if the same tests were run on a different JVM, the JRuby Mongrel speed might be faster than the C MRI Ruby speed.
Jruby 1.1.4 + Tomcat 6.0.14 / Glassfish 9.1u2
| Environment | Rails (rps) | Merb (rps) | Merb head (rps) | Notes |
|---|---|---|---|---|
| Glassfish production | 130 | 110 | 185 | Couldn’t get JRuby thread pool working. |
| Tomcat production | 144 | 75 | 197 | Couldn’t get JRuby thread pool working. There was some Tomcat/Merb specific problem here that was fixed in Merb head. |
Results: Rails is about 18% faster than Merb running on Glassfish. Rails is about 2x faster on Tomcat, though I expect that there is some sort of Tomcat-JRuby-Merb-specific bug here.
Merb head is running ~36%+ faster than Rails in Tomcat and Glassfish. The previous results where Tomcat was running 2X slower are now gone, as the Merb fixes eliminated that discrepancy.
Conclusion
In these tests, Rails is running around 9-18% faster than Merb in general (with the exception of Tomcat, which should be looked into). This implies that Merb has a little more overhead than Rails does, which is a surprise. Merb should have lower overhead, so why is it showing up slower in these tests?
With the updates by the Merb team, Merb is now running 36-70% faster than Rails, depending on the environment.
7 Comments
Hey Paul,
Those are very interesting numbers that don’t match any of the benches we’ve ever done. I’m going to investigate what the difference is, but I suspect that at least part of it is using AR, which we don’t usually do in our benches (our tests are about the core performance of the dispatcher and renderer, since we don’t ship with an ORM).
But still, that doesn’t full explain the results. I’m going to try to understand exactly what the difference is.
awesome! thanks for the udpate
Can you explain why you are using Merb head and Rails 2.0.2 ? Why not repeat with Edge Rails in all fairness ?
Considering, Edge rails is threadsafe ?
Peter, I’m using merb head here because the merb team specifically went in and fixed the bugs that were causing the slowdown. In addition to your request to benchmark edge rails, I’ve also been asked to benchmark Rails 2.1, and redo everything with different jruby optimization flags. As I have time, I will add all those environments to the table.
Edge rails being thread-safe shouldn’t make a difference here (except, perhaps, to slow it down slightly) because I’m not testing with any concurrency.
I am also interested in seeing the edge-rails results. However, unlike Rails, when Merb is slow, it’s usually the result of a tiny mistake that’s easy to fix (this slowdown was the result of a single, small commit that was easy to revert). Rails, on the other hand, has its performance issues very spread out.
That said, they have been adding performance improvements en masse, so I’d be interested in seeing it. I’d love a little speed war to heat up :)
Is there any particular reason you didn’t want to test concurrent connections?
4 Trackbacks
[...] great write-up by Paul Boone of various configurations using both MRI and JRuby, different versions of Rails, and [...]
[...] http://www.mindbucket.com/2008/09/18/rails-merb-performance-comparison-on-mongrel-jruby-tomcat-glass... Merb The Super Bike Of Frameworks View SlideShare presentation or Upload your own. (tags: ruby rails) [...]
[...] and JBoss AS 5, it would be sane to consider and evaluate JRuby to co-exist with Java. Also as Merb is faster running on JRuby and Merb will join forces with Rails to become Rails 3.0 would be another reason to evaluate JRuby [...]
[...] and JBoss AS 5, it would be sane to consider and evaluate JRuby to co-exist with Java. Also as Merb is faster running on JRuby and Merb will join forces with Rails to become Rails 3.0 would be another reason to evaluate JRuby [...]