Using Varnish to protect Apache against slowloris

From TykWiki
Jump to navigationJump to search

In June 2009 a tool called slowloris was created that makes it trivial for any script kiddie to tie up all the available connections on Apache and other web servers.

I downloaded the script and sure enough, I was able to take down all the Apache servers I manage (including the one you are reading this on) from an ADSL line (the servers are on much bigger links). So while this attack has been known for a long time, the fact that slowloris has been released makes it likely in my opinion that this will be exploited in the near future.

I set out to find an effective way to mitigate the attack, and Varnish came up as a solution among many others suggested around the web.

  1. Tweaking Apache timeout settings
  2. Using the accf_http kernel module has been suggested as an effective mitigation
  3. pf overload techniques has been suggested (say, block an IP if it opens more than 50 connections in five seconds to the Apache port)
  4. Setup lighttpd or Varnish infront of the Apache server, and proxy only valid requests.

1. I tried tweaking Apache timeout settings, but I couldn't come up with a setting where I couldn't still take down the server, at least for a short while.

2. The accf_http kernel module for FreeBSD effectively mitigates the attack when using GET requests - but if you give slowloris the -httpready parameter it switches to POST, which bypasses the accf_http filter completely.

3. pf overload statements can be somewhat effective at mitigating the attack from a single source, but this attack will probably become distributed at some point in which case I am back to square one. Besides, users behind a proxy/single-ip firewall might easily make a few 100 connections from the same IP, with completely legitimate surfing.

4. I read somewhere - I can't remeber where, and I apologize if it isn't true - that lighttpd could be vulnerable to this attack in some configurations. Also, out of the two, Varnish is the tool designed for the task, it was actually build to be placed infront of a webserver.

So I decided to go with varnish. This article is about the Varnish configuration and the changes I made to my Apache config.

Setting up Varnish to protect Apache

First step was installing Varnish and getting it up and running.

Configuring Varnish

This was relatively simple, after installing Varnish from /usr/ports/www/varnish I made a blank configuration file and added the following VCL to it:

[tykling@harmony ~]$ cat /usr/local/etc/varnish/tyktest.vcl
# Define a backend, Apache is now listening on
backend tyktest {
        .host = "";
        .port = "8080";

# Define what to do when receiving a request;
# - Send all requests to the backend "tyktest"
# - Remove X-Forwarded-For if it is already set for some reason
# - Add a new X-Forwarded-For header, set it to the clients IP (for logging on Apache)
sub vcl_recv {
        set req.backend = tyktest;
        remove req.http.X-Forwarded-For;
        set    req.http.X-Forwarded-For = client.ip;

# Set timeout for all objects to 0 seconds. This defeats the purpose of running a cache,
# but this is for the added security only. 
# This was the easiest way for me to preserve logging of all requests.
sub vcl_fetch {
   set obj.ttl = 0s;

The options are explained in the comments above them. I also added the following to rc.conf:

[tykling@harmony ~]$ grep varnish /etc/rc.conf

These are fairly self-explanatory. The varnishd_listen directive states that Varnish should listen on all addresses on port 80. You can of course limit this to only one address if you want to. The varnishd_storage directive tells varnish where to keep it's cache - this doesn't really matter for my purposes since I disabled all caching. But the folder should exists and be owned by www:www to keep Varnish happy (the default user Varnish runs under is the same as the default user for Apache, conveniently).

Configuring Apache

A few changes is neccesary in the Apache config to make this work. First I changed the port Apache listens on, so I changed the following things in httpd.conf:

[tykling@harmony ~]$ grep -A 2 varnish /usr/local/etc/apache22/httpd.conf
#varnish change
#varnish change

I also changed the NameVirtualHost directive in my httpd.conf:

[tykling@harmony ~]$ grep -A 3 "Virtual hosts" /usr/local/etc/apache22/httpd.conf
# Virtual hosts
#varnish change
#NameVirtualHost *:80
NameVirtualHost *:8080

And finally I changed the first line of all my vhost configs from:

<VirtualHost *:80>


#varnish change
#<VirtualHost *:80>
<VirtualHost *:8080>

One important thing remains. The Apache logging is currently set to something like this in my Apache configs:

CustomLog "|/usr/local/sbin/cronolog --symlink=/usr/local/www/logs/ /usr/local/www/logs/" combined

Don't be alarmed by the fact that I am using cronolog to log with, a more normal logging statement without cronolog could look like:

CustomLog "/usr/local/www/logs/" combined

The combined logging format is (from the makers of Apache) defined in httpd.conf:

[tykling@harmony ~]$ grep combined /usr/local/etc/apache22/httpd.conf
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

I defined a new format called varnishcombined as suggested on the Varnish site:

[tykling@harmony ~]$ grep varnishcombined /usr/local/etc/apache22/httpd.conf
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined

This tells apache to log the X-Forwarded-For IP instead of the request IP which is since Varnish is running on the localhost.

I then proceeded to change all my vhost CustomLog statements to use the new varnishcombined format instead of the standard combined.

Starting everything up

I restarted Apache and started Varnish in that order, and everything seems to work smoothly. Every request is logged in the Apache logs, with the correct IP address, which is exactly what I wanted. Varnish listens on port 80, receives the (entire, which is important in this context) request, and forwards it to port 8080 where Apache is ready to hand out content. And best of all, I am no longer able to take down the server with slowloris.

--Tykling 18:54, 24 June 2009 (UTC)