LiteSpeed vs Apache vs Nginx, Who Does Magento Hosting Better?

Before last years Magento Imagine conference, we announced a benchmark where we stated that LiteMage performed ~17-40x better than Turpentine + Varnish. This led to several people questioning the validity of our benchmark because LiteMage seemed too good to be true. A year later, we decided to revisit this benchmark with the latest version of each software to see where we stand.

In this benchmark, we compare Magento 1.9 running on LiteSpeed Web Server, Apache, and Nginx to find out which web server better meets the needs of the modern eCommerce environment.
The Challenge

We tested our 3 server setups both with and without a caching solution by running siege to simulate 50 simultaneous users accessing a Magento store. In an attempt to resemble a real-world production environment, we had each simulated user access 10 random catalogue pages from a list of 127 items.
The Contestants

LiteSpeed Web Server 5.0.13 with LiteMage Cache 1.0.16
Nginx 1.8.1 with Varnish Cache 4.0.3 + Turpentine 0.6.8
Apache 2.2.15 with Varnish Cache 4.0.3 + Turpentine 0.6.8
LiteSpeed Web Server 5.0.13
Nginx 1.8.1
Apache 2.2.15
The Configuration

Server
2 cores of Intel(R) Xeon(R) CPU E5-1620 Quad Core @ 3.60GHz
4G RAM
Running CentOS 6.7 (final)
Intel X540 10GBASE-T on board NIC

Client
2 cores of Intel(R) Xeon(R) CPU E5620 Quad Core @ 2.40GHz
4G RAM
Running Debian GNU/Linux 8 ( Jessie )
On board Intel 82599EB 10 Gigabit Ethernet Controller

Network
Netgear XS708E-100NES 8-ports 10G switch

Varnish Configuration
Varnish has been configured to use malloc as its storage method. The varnish command we used is:
/usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f /etc/varnish/default.vcl -T 127.0.0.1:6082 -t 120 -p thread_pool_min=50 -p thread_pool_max=1000 -p thread_pool_timeout=120 -u varnish -g varnish -S /etc/varnish/secret -s malloc,512M -p cli_buffer=16384 -p feature=+esi_ignore_other_elements -p vcc_allow_inline_c=on

The Data

For this benchmark, we used Magento 1.9’s sample data set. Our list of test URLs is generated from the sitemap and contains 127 entries.
The Test Methods

Regardless of whether testing LiteMage or Turpentine + Varnish, we followed the same procedure.

1. Install the Magento extension, configure it, and use a browser to verify that cache is working.
2. Warmup the cache with the following command:
   siege -b -c 1 -r 127 -f urls_list.txt
3. Check a few URLs randomly with a browser to make sure a “Cache: hit” response header is in the response.
4. Run the benchmark test to simulate 50 clients accessing 10 random cached pages.
   siege -b -c 50 -r 10 -i -f urls_list.txt
5. Run this test multiple times and use a result that falls close to the average.

The Results: 50 Users Each Accessing 10 Random Pages

Requests Per Second

While LiteSpeed + LiteMage clearly outperforms the competition by a wide margin here, it is important to note that regardless of which web server you choose, in this regard, caching makes a huge difference, in the worst case more than tripling the amount of requests per second on average.

Longest Transaction Time

Again, in this measurement, LiteSpeed + LiteMage blows away the competition. In general, longest transaction time also improved for every server when using cache, but by a much slimmer margin. Web servers using Varnish cache tended to only improve by ~10% but LiteSpeed Web Server cut nearly 90% of the time its longest transaction took while using cache.

Throughput

In a well-performing application, network bandwidth is not a limitation. These tests used a 10Gb switch and 10Gb cards on both the client and server machines. These results are based on software limitations.

This final measurement is yet another example of how caching puts your web server on an entirely different level. While LiteSpeed + LiteMage outperforms the competition yet again, the throughput for every server improved, in most cases by more than 5x, when using cache.

We measured how fast your web server performs when serving random pages to 50 users at the same time. In this regard, LiteSpeed + LiteMage is the clear standout among all included server setups. If you’re running a Magento store, you want to run it with a caching solution, and there’s no caching solution faster than LiteMage cache!
Bonus: 100 Page View Test

Some of you might doubt our test results, particularly because you may get much better performance out of Varnish. The reason for this difference is probably simple. Perhaps during our testing we used a different testing tool or testing methods and this change has had a huge impact on the results of our test.

Our advice is to NEVER use “Apache Bench” or “ab” for this type of testing. “siege” is capable of following cookies while “ab” cannot. Using commands like

ab -n 500 -c 50 http://your.test.site/

does not simulate traffic for 50 visitors with each visitor accessing 10 pages. It actually simulates “ONE visitor making 10 parallel connections and accessing the same page 500 times.” The result of this command may look stellar, but it in no way resembles a real world scenario and could not be more unrealistic.

We suggest sticking with “siege” unless you have access to more advanced testing tools such as “loadrunner”.

Even with “siege”, the number of page views for each visitor has a huge impact on the result. Here are the results of running the above tests with 100 page views per visitor. Varnish gets much better results in this round of testing, but you should consider the actual value of these results. In a real world situation, how many visitors will access 100 catalog pages in your store?

These results can be seen below.

For more details visit our Magento + LiteMage Benchmark page!

If you have any feedback/comments/recommendations, we are keen to hear from you. Shoot us an email at: info@litespeedtech.com