Performance Comparison: WP Super Cache and W3 Total Cache

The Setup

I wanted a relevant performance comparison of the two most popular (at the time) caching solutions for WordPress before I chose one for this blog.

In my opinion, there are two key issues to getting relevant performance comparisons and benchmarks.

The first is the hardware and software involved. Different versions can have huge differences in performance profiles so you want test with as close a match as you can get.

I was not able to match the hardware my live server uses, so I was inclined to make sure the software was as closely matched as humanly possible.

I installed Ubuntu Server 8.04 and the latest PHP (5.2.4-2ubuntu5.17) into my virtual machine and brought it all up to date. I then configured Apache (Apache/2.2.8 for the curious) to match the configuration of my blog's server (which modules were loaded, php via fastcgi, APC installed, the php.ini values etc). The same was done for the MySQL instance.

After confirming everything was working I installed the latest WordPress release (at the time version 3.1.3) and the importer plugin and imported the WordPress sample data. I then shut down the virtual machine and enabled the "undo disk" feature.

This brings me to the second issue when looking at performance, which is repeatability. Test results are skewed by changes to the environment (e.g. extra processes running, or extra network traffic) so it's important to (preferably) eliminate or baseline those external factors.

The "undo disk" feature proved very helpful in my efforts to minimize the influence of those external factors.

The Methodology

As these tests were being run in a virtual machine I couldn't expect similar numbers as my live site. However, as I'm looking to compare the two solutions, relative performance to one another is all that's really important. I decided to run apachebench from within the virtual machine to eliminate a lot of issues that would be out of my control (communication between the host machine and the virtual machine, the host using additional cpu time, etc).

This was not ideal as I'm stealing potential resources from the apache instance, but compared to the alternatives it was the best option. After running a few tests and looking at the numbers I decided on 3 runs of 1000 requests each at both a concurrency of 1 and a concurrency of 5. The concurrency level of 5 was enough to generate some contention for resources but not enough to bog down the virtual machine.

The command line looked something like this

ab -n 1000 -c 5 http://localhost

After each run of tests I shut down the virtual machine and cleared the undo disks to restore the machine's baseline state.

After each boot the virtual machine was given an equal amount of time to "settle down" before the tests were run.

The Results

For each of the results I calculated averages of the 3 "c1" and 3 "c5" runs as well as an overall average across all 6 tests. The values in the rows are requests per second.

To get an idea of what the virtual machine was capable of doing, my initial tests were against a static html file (the "it works" index file). The results are as follows

The next test was to determine how well it fared with php involved. I chose the "phpinfo" page as my test subject.

With a server baseline in hand, it was on to WordPress.

For the WordPress tests I opted to test a few different code paths. I tested the following scenarios:

• Front page: http://localhost/
• Hello World: http://localhost/2011/06/hello-world/
• Archive Page: http://localhost/2008/06/
• Category Page: http://localhost/category/uncategorized/
• Tags Page: http://localhost/tag/post-formats/

With only itself to rely on, the WordPress results are not pretty.

I again reset the machine and installed WP Super Cache (v 0.9.9.9) and W3 Total Cache (v 0.9.2.2). I did not enable or configure either of them however, I simply made the point in time after their installation my new "reset" point to save myself from having to do the installation before each test.

Up first was WP Super Cache. In the first set of tests, I simply enabled the plugin. This enabled caching via PHP, which while a noticeable boost could definitely be improved upon by modifying the configuration.

For the second set of tests WP Super Cache was configured to use mod_rewrite with 304 redirects and compression enabled. This was a huge boost and more in line with what I'd been expecting.

W3 Total Cache has considerably more options than WP Super Cache, and was the focus of the rest of the tests. The inital test was again just to enable the plugin. For W3 Total Cache this equates to using "disk enhanced" for Pages and enabling minify to use "disk". These two options were standard for the rest of the tests

For the second set of tests with W3 Total Cache the object and database caches were enabled for "disk" as well. Surprisingly this had an adverse effect on performance.

For the third set of tests, I decided to leverage APC for the object cache and disable the database cache. This didn't appear to make much of a difference from the second round.

For the final set of tests, I reversed my choice from the previous tests and set the database cache to use APC while disabling the object cache. Again, not much of a difference among the last three rounds of tests, and none performed as well as the initial set of tests that didn't leverage the database or object cache.

I was surprised to see that the database and object caches were negatively impacting performance, but upon reflecting it seemed the likely cause was the virtual machine.

The Conclusion

Both caching solutions at their best offer comparable performance. If you're looking at pure requests per second, WP Super Cache came out on top in my setup, but only when it was configured to leverage disk caching.

W3 Total Cache has more options in regards to how you cache items, includes advanced support for using a content delivery network (or cookieless subdomain) and should definitely be a consideration if you're looking for a more technically advanced caching solution.

That said, WP Super Cache is easy to setup and maintain and works really well so it should probably be the default choice for most people's caching needs.

I am a tinkerer however, so I ended up choosing W3 Total Cache (as you can see by examing the source generated for this page) so that I could tweak it until I was content.

Follow Up

After setting up W3 Total Cache on my blog, I found some "quiet time" to do additional testing.

On my live server I did get about a 5% performance boost per cache (database and object) configured to use APC, so I left them enabled for an easy 10% gain. This reinforces my point earlier about testing on a system as close to what you're deploying to as possible if you want accurate results.