Gameboy Genius

500 infernal terror!

My webhost recently started moving their customers to their new generation of servers, running LiteSpeed. This has caused me some grief the last few hours. One of my sites consistently showed a “403 access denied” error, seemingly no matter what I tried to do to remedy the problem. With some hard trying, I managed to make it display a “404 not found” instead, still seemingly randomly.

It turns out there were two problems, both most likely caused by buggy interpretation of .htaccess files by LiteSpeed. The first problem came from these two lines:
Allow From All
Order Allow,Deny
As it turns out by experiment, LiteSpeed handles these statements differently from Apache. Apache will start off by making a list of all the applicable rules, and then evaluating them in the order specified in the Order statement. In this example, first all Allow rules will be evaluated, then all deny rules. If there are any matches, Deny will take precedence since it’s last in order. If there are no matches, Deny will also take precedence since it’s the second argument. Of course, in this case everyone will be allowed since there’s an Allow From All statement, unless there are matching Deny rules. This is well documented in the Apache manual and (hopefully) well understood by web developers.

LiteSpeed on the other hand is dependent on where statements are placed in the source code. In the above example, the Allow From all statement is ignored, or possibly overridden by some implicit form of Deny From all. Not really sure… Moving Allow From all to after the Order statement will make it work. Also, changing the Allow statement to something more specific, like Allow From 1.2.3.4 but still placing it before Order will suddenly make it trigger. Not sure what’s up with that. Perhaps the LiteSpeed authors are trying to make it work how they think people think it will work. Or maybe they’re trying to do some optimizations by trying to abort the evaluation of the file as early as possible. Or maybe they just don’t know how these rules are interpreted by Apache. In any case, fact of the matter is that it doesn’t work as it is supposed to.

The next problem is how LiteSpeed handles errors in .htaccess files. It seems like whatever you do, you can’t get a 500 internal error from LiteSpeed if your .htaccess rules are incorrect. Completely invalid rules will be ignored, whereas as they would give a 500 internal on Apache. Malformed rules will give you a 404 not found instead of a 500 internal error, making it difficult to pinpoint the exact error. I’m not sure if this is a “feature” of LiteSpeed so they can say that their server will never have internal errors, or if it’s a secondary error similar to the “Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request” errors that you sometimes see on badly configured Apache servers.

The culprit this time was the following rule:
RewriteRule ^(.*)$ wiki/index.php?title=$1 [PT,L,QSA,B]
LiteSpeed apparently doesn’t support the B modifier (escape characters like & when building the query string) which caused the server to barf. However, since what I got was a 404 error, and not a 500, I thought the rule somehow rewrote the URI to something that didn’t exist. It was not until I replaced the B with a bunch of random characters that I understood what was going on.

All in all, all of this may not seem like a very tough nut to crack, but there’s more. For some reason, an index.html file had its permission mask set to 400, or r——–, which in UNIX persmission terms means that the file will only be readable by its owner, and not by users in the same group, or “anyone”. This was a placeholder page that the web host had put their, but I’m not sure how it got that permission mask.

When the rule works as it should, the root URL http://packetsofknowledge.se/ should be rewritten to the MediaWiki index.php. However, when I tried simply deleting/renaming my .htaccess, I still got a 403 because of the permissions of the index.html. However, since I hadn’t noticed that, I assumed at that point that the issue maybe wasn’t .htaccess related.

To summarize, when you have different error sources, the number of ways something can fail increases exponentially.