After making our app thread-safe and transitioning to Puma, I’ve done a bit of benchmarking to see what effect, if any, the new server has on performance.
Using Apache Bench, we can gather basic information about how our application responds under stress both before and after the transition. Since it was important to me to understand how the server would take the load before making any hard-to-undo changes to the production environment, the following tests were run on our staging server. It should have similar performance characteristics to our production servers, except it has less RAM (2GB instead of 4).
I configured both Unicorn and Puma to use two workers (processes). Puma additionally allows you to configure the minimum and maximum numbers of threads it will spawn, and I was interested in seeing how this might affect throughput. As shown below, I tested 3 different Puma configurations – a maximum of 1, 2, and 16 (the default number) of threads.
ab commands used for testing took the following form:
ab -n 100 -c 10 127.0.0.1:3000/mic-test-1scbld4eA7KfCFESbEH2h1oDY7U
-n 100 indicates that we want 100 total connections to be made, and
-c 10 tells
ab to attempt 10 connections at a time. This is the number I change to test different concurrency levels.
Median response time was statistically unchanged across all server variations. Interestingly, 99th-percentile performance is consistently better in Unicorn-land, especially when compared to the single-threaded Puma run, the closest thing to an apples-to-apples comparision.
Although the MRI global interpreter lock (GIL) fundamentally limits what you can do with concurrency in the default Ruby implementation, I have to admit that I had hoped for more visible improvement with a multithreaded application server, especially with a page that calls out to the database and node.js, actions that should release the GIL while they’re being processed. It just goes to show that there is no excuse not to test when making changes that could substantially affect the performance of your app.
Long term, our plans are still to switch to JRuby when JRuby 9000 comes out (hopefully in a few months). At that point I’ll repeat these tests, and with a bit of luck its true concurrency will allow for a substantial performance improvement.