I've been administering Wordpress blogs for almost a decade now; hosting for ... 5? 6 years? I'm not sure any more.
Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module.
My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable.
So now I'm on Nginx. And with some fancy footwork in the rewrite rules to support WP Super Cache, it can serve static files and gzipped HTML straight off disk.
Unsurprisingly, in this configuration, it runs like the clappers.[1][2]
One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later.
[1] At least, it used to do this, until I updated everything and it silently stopped working the way it used to (which I only discovered while outlining my current config in comments below). I mean, thanks to memcached and closer servers and running Percona on a separate server it's still pretty fast, but this change bugs me and you can bet I'll be fixing it later today after work.
[2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode cache.
The query cache because I don't think it buys me enough versus using memcached (just a gut feel, I've got no hard numbers to back me up) and because it often turns into a contention point in read-heavy MySQL instances. Also, by dropping it I can free up more memory for InnoDB.
The opcode cache because they tend to be flaky. I've had bad experiences with all three major ones. If you ever pop Wordpress up on a profiled PHP instance you find that its time is spent overwhelmingly on waiting for MySQL, concatenating strings and then streaming it out to the webserver, for which proper on-disk/in-memory caching is the answer. Time spent loading and compiling .php is minimal by comparison, so why put up with the hassle?
I don't use varnish because the whole philosophy is to let the OS select what to cache in memory, and ... well I already do that. Plus getting ESI to work on frequently updated objects like "recent comment" widgets is a hairy pain in the arse and I just can't be bothered.
My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable.
It is not a bug, it is a feature. lighttpd used to wait 60s before checking the backend again. Nowadays the default is 1s. Set disable-time to 0 if you don't like it (it should be the default IMHO.)
In case you're wondering whatever happened to this bug, it's still happening! I installed the latest release of Lighttpd a few weeks ago and it happened on one of our sites. So amazing to see the same bug still kicking after 3-4 years..
The majority of our transient bugs on WordPress.com are probably related to APC opcode issues. We try to catch them and handle it gracefully, but it isn't always so easy.
I'm interested to hear what versions of PHP and APC you used.
In my experience, stable APC releases have been very stable, but beta APC can poop out pretty bad, even if the changelog doesn't indicate anything that might affect your application.
I've used stable APC + PHP + Apache releases to do some large things, so not sure why APC is an utter bomb in 2012.
No kidding. There really aren't "three major opcode caches" anymore, APC won out years ago and is now tightly integrated with PHP compiles and as a module available in any package manager.
It's worth revisiting. We had huge problems with it when we tried it out in 2007, but I tried it again earlier this year and was pleasantly surprised - I got everything working, easily, and nothing broke. And I got that speed up.
My gut feeling tells me that your profiling is correct, however - any significant PHP application (that isn't written terribly inefficiently) will have its bottleneck in data access and not the parse/compile stage. There's very little point to speeding up by 250% a portion of your app that only accounts for 2% of execution time!
I'd be interested to read on your blog about what APC problems you've encountered - we have many wp installs with APC and the only problem is it will segfault once in a blue moon but we can automatically restart when that happens.
You mentioned opcode cache problems - some people complain of segfaults and I am just pointing out you can make it restart if that ever happens. (I've had it trigger three times this entire year so far.)
There is most certainly a serious load reduction when using an opcode cache. It's not just common sense, you easily find a dozen independent benchmarks on the web proving it.
Now if you had the opcode cache running incorrectly you might not see the improvement (ie. some people misconfigure it where the memory is not shared and persistant, and instead gets destroyed as php children are created/removed).
I've never heard of anyone not get incredible speedups with APC. We're talking 250%+. Every (large) production PHP deployment I've seen, disabling APC would make everything fall over. They would need 3-4x as much capacity.
I think the big point here is that on a Wordpress site, most hits don't actually get as far as PHP -- most of the time, pages can be served statically from wp-supercache or similar.
"One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later."
Edit: actually, it looks like this doesn't work the way I think it does ... looks like nginx has changed enough between 0.7 and 1.0 that my original config doesn't work as correctly as it used to.
Edit 2: Actually I think WP Supercache is the one that changed.
I particularly liked how they used the try_file directive to cut down on the if statements.
----
Like most people I've cobbled together what I found through Google searching. My case is further complicated because I run a multisite configuration and use the WP Domain Mapping plugin.
It works in a few stages.
WP Supercache is configured to gzip pages to disk. It used to be that you needed to add extra code by hand to support domain mapping, but WP Supercache now cooperates and puts on-disk cached files for each blog in its own directory.
Then I put specific directives in a sites-available/ config file, not the main Nginx file (I serve a static site off the same server).
First,
server {
listen <ipaddress> default_server;
The default_server directive means that if another listen directive doesn't pick up an incoming request, it will be handled by this config. Otherwise multisite goes kerflooie.
Getting end-to-end UTF8 on a stack is a hassle because you have to do it in the database (multiple times, MySQL has about two hojillion charset configuration dials), in PHP and then in Nginx:
The rewrite is just a little tweak. If you go to /wp-admin, Wordpress will often redirect to the home page. If you go to /wp-admin/, it does what you expect. So this rewrite just adds a trailing slash.
index index.php;
For multisite, the Wordpress coders change the URLs files are served from, for reasons that are simply beyond my mortal comprehension. Anyhow, you need a rewrite rule for /files/ URLs:
Then what follows is based on the common config file you'll see on a dozen blog and forum posts if you google around.
The next thing to do is try and serve a file straight off disk. Nginx does things in an unintuitive order, so even though this rule is further down, it will often fire first:
# Rewrite URLs for WP-Super-Cache files
# if the requested file exists, return it immediately
# this covers the static files case
if (-f $request_filename) {
expires 15d;
break;
}
The "break" directive basically says, "Oh you found whatever.css|jpg|js? Just serve that up kthxbai".
Now we proceed to determine whether or not we'll try to serve a cached file off disk:
set $supercache_file '';
set $supercache_uri $request_uri;
# don't interfere with POST events (ie form submissions)
if ($request_method = POST) {
set $supercache_uri '';
}
# Using pretty permalinks, so bypass the cache for any query string
if ($query_string) {
set $supercache_uri '';
}
# Don't show cached version to logged-in users
if ($http_cookie ~* "comment_author_|wordpress|wp-postpass_" ) {
set $supercache_uri '';
}
If there's still a supercache URL, we try to serve it straight off disk:
# if we haven't bypassed the cache, specify our supercache file
if ($supercache_uri ~ ^(.+)$) {
set $supercache_file /wp-content/cache/supercache/$http_host/$1index.html;
}
# only rewrite to the supercache file if it actually exists
if (-f $document_root$supercache_file) {
rewrite ^(.*)$ $supercache_file break;
}
Otherwise, give up and let Wordpress grind out the file:
# all other requests go to Wordpress
if (!-e $request_filename) {
rewrite ^.+?(/wp-.*) $1 last;
rewrite ^.+?(/.*\.php)$ $1 last;
rewrite ^ /index.php last;
}
}
location ~ .php$ {
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /var/www/wordpress$fastcgi_script_name;
include /etc/nginx/fastcgi_params;
}
From the Wordpress wiki I picked up a few nifty directives to make log files a little less cluttered; I've thrown those in a common file which I include here:
include /etc/nginx/clean.conf;
}
clean.conf looks like this:
# Global restrictions configuration file.
# Designed to be included in any server {} block.</p>
location = /favicon.ico {
log_not_found off;
access_log off;
}
location = /robots.txt {
allow all;
log_not_found off;
access_log off;
}
# Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
# Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
location ~ /\. {
deny all;
}
# Deny access to any files with a .php extension in the uploads directory
# Works in sub-directory installs and also in multisite network
# Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
location ~* /(?:uploads|files)/.*\.php$ {
deny all;
}
The final step is to add this magic directive to your master nginx.conf:
gzip_static on;
This tells Nginx to look for a file with a .gz extension. So when it gets redirected to look at whatever-blog.com/index.html, it will also check for index.html.gz. If it finds that file, it will serve it instead.
Googling is about the worst way to set up an nginx server configuration! The only tutorials and helpful snippets out there are riddled with bad practices.
I notice that you're using a lot of "if" statements. I hope you've read this http://wiki.nginx.org/IfIsEvil. I understand that in this case it might be harder to find alternatives for the if-statement, haven't looked into what you need really deeply.
It's been a few years since I cobbled this one together and I recall reading the IfIsEvil web page. I vaguely recall that at the time there was some limitation on how try_files worked that made it necessary to use ifs, but for the life of me I don't remember what.
The rtCamp configuration I linked to earlier makes use of try_files and says he has support for both supercache and domain mapping, so I think I'll migrate to that.
Thanks Jacques. Author of that updated config post on rtCamp.com here.
I really think Nginx in still underestimated by large. Most article handles rewrite in nginx based on their Apache knowledge.
Nginx's try_files is magical. So does maps{..} section.
Using Nginx map, you can server static files in WordPress-multisite without PHP (much better than X-SendFile or X-Accelredirection). On large wordpress-multisite network, this can really increase Nginx's capacity by multifold.
It's slightly nicer than stock MySQL and I can more easily get Percona to help me if it goes bang (it has far more stats available than stock MySQL, for instance).
Some of my sites have sufficient write activity that using XtraDB (basically a slightly souped-up InnoDB) is a smarter option than MyISAM. For searching people can just use Google, it does a much better job than MySQL's inbuilt fulltext search.
I don't know if they've fixed it, but AFAIK tables with full text fields used to cause MyISAM joins to go to disk, even if the full text field isn't included in the query. As you can imagine, this sucks. Maybe that's been fixed.
We started using Percona's MySQL builds by default a couple of years ago. There are some nice performance and convenience features not included in the MySQL.com builds. (Un)fortunately we still have quite a few MySQL 4.1 instances happily running and are still using MyISAM across over 360 million tables. Most of that stuff is not running Percona.
Not sure in this environment, but I've been using Percona builds of MySQL for the last couple years and it's a great drop-in. Good optimizations for InnoDB Oracle-owned bits.
When I first read about Varnish it was framed as a "correct" implementation of what Squid was trying to do. It seems like people are using Varnish in situations where they would not otherwise be using Squid.
Varnish leaves it to the OS to decide what pages to keep in RAM and which to leave on disk; essentially to avoid the double-buffering problem and because the OS has a larger view of the total machine's requirements.
But with wp-supercache I already have that approximate architecture. Files are on-disk, Nginx selects them, the OS notices that some files are frequently accessed and silently caches them in RAM. Everyone wins.
And I don't need to add ESI directives to my themes to get Varnish to handle frequently-updated material correctly.
Anyhow. I started with Apache. Apache probably works fine if you load the mpm-oh-you-didnt-know-about-mpm-so-sorry module.
My next stop was lighttpd. A fine piece of software, with a persistent bug that caused it to drop into a 500 state if FastCGI instances were ever unavailable.
So now I'm on Nginx. And with some fancy footwork in the rewrite rules to support WP Super Cache, it can serve static files and gzipped HTML straight off disk.
Unsurprisingly, in this configuration, it runs like the clappers.[1][2]
One thing that always surprises me is seeing configurations where Nginx is merely a proxy or load balancer into a bunch of Apache instances. Are you nuts? Stop doing that. Just use Nginx and FastCGI -- thank me later.
[1] At least, it used to do this, until I updated everything and it silently stopped working the way it used to (which I only discovered while outlining my current config in comments below). I mean, thanks to memcached and closer servers and running Percona on a separate server it's still pretty fast, but this change bugs me and you can bet I'll be fixing it later today after work.
[2] Two optimisations I've dropped are the MySQL query cache and a PHP opcode cache.
The query cache because I don't think it buys me enough versus using memcached (just a gut feel, I've got no hard numbers to back me up) and because it often turns into a contention point in read-heavy MySQL instances. Also, by dropping it I can free up more memory for InnoDB.
The opcode cache because they tend to be flaky. I've had bad experiences with all three major ones. If you ever pop Wordpress up on a profiled PHP instance you find that its time is spent overwhelmingly on waiting for MySQL, concatenating strings and then streaming it out to the webserver, for which proper on-disk/in-memory caching is the answer. Time spent loading and compiling .php is minimal by comparison, so why put up with the hassle?
I don't use varnish because the whole philosophy is to let the OS select what to cache in memory, and ... well I already do that. Plus getting ESI to work on frequently updated objects like "recent comment" widgets is a hairy pain in the arse and I just can't be bothered.