Using Varnish to protect Apache against slowloris: Difference between revisions

From TykWiki
Jump to navigationJump to search
No edit summary
Line 110: Line 110:


=== Apache client IP 127.0.0.1 problem ===
=== Apache client IP 127.0.0.1 problem ===
One important thing remains: All requests seen by Apache now look like they come from 127.0.0.1. This needs to be fixed in the logs obviously, so I can look at [http://awstats.sourceforge.net/ awstats] when I have a hangover. But more importantly (or less, depending on the content Apache is serving) the other Apache modules (like mod_php) also see 127.0.0.1 as the client IP. If you are running an application that somehow uses $_SERVER['REMOTE_ADDR'] then the application might run into problems when all clients suddently have the same 127.0.0.1 remote address. If this is of no importance to you, like if you are only serving static HTML content, you ''can'' fix just the logging. On the other hand, fixing the problem completely is easier than just fixing the logging, although it does require the installation of an additional Apache module. Good thing we have ports.
One important thing remains: All requests seen by Apache now look like they come from 127.0.0.1. This needs to be fixed in the logs obviously, so I can look at [http://awstats.sourceforge.net/ awstats] when I have a hangover. But more importantly (or less, depending on the content Apache is serving) the other Apache modules (like mod_php) also see 127.0.0.1 as the client IP. If you are running an application that somehow uses $_SERVER['REMOTE_ADDR'] then the application might run into problems when all clients suddently have the same 127.0.0.1 remote address. If this is of no importance to you, like if you are only serving static HTML content, you ''can'' fix just the logging. On the other hand, fixing the problem completely is easier than just fixing the logging, although it does require the installation of an additional Apache module. Good thing we have ports to make that part easy.
 
To recap, you need to do one of two things:
* Fix the Apache client IP problem with mod_rpaf2 (recommended)
'''or'''
* Fix the Apache client IP logging only (not recommended)
 
Continue with the relevant section below.


==== Fixing the Apache client IP problem with mod_rpaf2 (recommended) ====
==== Fixing the Apache client IP problem with mod_rpaf2 (recommended) ====

Revision as of 14:17, 7 July 2009

In June 2009 a tool called slowloris was created that makes it trivial for any script kiddie to tie up all the available connections on Apache and other web servers.

I downloaded the script and sure enough, I was able to take down all the Apache servers I manage (including the one you are reading this on) from an ADSL line (the servers are on much bigger links). So while this attack has been known for a long time, the fact that slowloris has been released makes it likely in my opinion that this will be exploited in the near future.

I set out to find an effective way to mitigate the attack, and Varnish came up as a solution among many others suggested around the web.

  1. Tweaking Apache timeout settings
  2. Using the accf_http kernel module has been suggested as an effective mitigation
  3. pf overload techniques has been suggested (say, block an IP if it opens more than 50 connections in five seconds to the Apache port)
  4. Setup lighttpd or Varnish infront of the Apache server, and proxy only valid requests.

1. I tried tweaking Apache timeout settings, but I couldn't come up with a setting where I couldn't still take down the server, at least for a short while.

2. The accf_http kernel module for FreeBSD effectively mitigates the attack when using GET requests - but if you give slowloris the -httpready parameter it switches to POST, which bypasses the accf_http filter completely.

3. pf overload statements can be somewhat effective at mitigating the attack from a single source, but this attack will probably become distributed at some point in which case I am back to square one. Besides, users behind a proxy/single-ip firewall might easily make a few 100 connections from the same IP, with completely legitimate surfing.

4. I read somewhere - I can't remeber where, and I apologize if it isn't true - that lighttpd could be vulnerable to this attack in some configurations. Also, out of the two, Varnish is the tool designed for the task, it was actually build to be placed infront of a webserver.

So I decided to go with varnish. This article is about the Varnish configuration and the changes I made to my Apache config.

Setting up Varnish to protect Apache

First step was installing Varnish and getting it up and running.

Configuring Varnish

This was relatively simple, after installing Varnish from /usr/ports/www/varnish I made a blank configuration file and added the following VCL to it:

[tykling@harmony ~]$ cat /usr/local/etc/varnish/tyktest.vcl
#
# Define a backend, Apache is now listening on 127.0.0.1:8080
#
backend tyktest {
        .host = "127.0.0.1";
        .port = "8080";
}

#
# Define what to do when receiving a request;
# - Send all requests to the backend "tyktest"
# - Remove X-Forwarded-For if it is already set for some reason
# - Add a new X-Forwarded-For header, set it to the clients IP (for logging on Apache)
#
sub vcl_recv {
        set req.backend = tyktest;
        remove req.http.X-Forwarded-For;
        set    req.http.X-Forwarded-For = client.ip;
}

#
# Set timeout for all objects to 0 seconds. This defeats the purpose of running a cache,
# but this is for the added security only. 
# This was the easiest way for me to preserve logging of all requests.
#
sub vcl_fetch {
   set obj.ttl = 0s;
}

The options are explained in the comments above them. I also added the following to rc.conf:

[tykling@harmony ~]$ grep varnish /etc/rc.conf
varnishd_enable="YES"
varnishd_listen=":80"
varnishd_config="/usr/local/etc/varnish/tyktest.vcl"
varnishd_storage="file,/usr/varnish,50%"

These are fairly self-explanatory. The varnishd_listen directive states that Varnish should listen on all addresses on port 80. You can of course limit this to only one address if you want to. The varnishd_storage directive tells varnish where to keep it's cache - this doesn't really matter for my purposes since I disabled all caching. But the folder should exists and be owned by www:www to keep Varnish happy (the default user Varnish runs under is the same as the default user for Apache, conveniently).

Configuring Apache

A few changes is neccesary in the Apache config to make this work.

Apache listen port

First I changed the port Apache listens on, so I changed the following things in httpd.conf:

[tykling@harmony ~]$ grep -A 2 varnish /usr/local/etc/apache22/httpd.conf
#varnish change
#Listen 10.16.0.70:80
Listen 127.0.0.1:8080
--
#varnish change
#ServerName 10.16.0.70:80
ServerName 127.0.0.1:8080

I also changed the NameVirtualHost directive in my httpd.conf:

[tykling@harmony ~]$ grep -A 3 "Virtual hosts" /usr/local/etc/apache22/httpd.conf
# Virtual hosts
#varnish change
#NameVirtualHost *:80
NameVirtualHost *:8080

And finally I changed the first line of all my vhost configs from:

<VirtualHost *:80>

To:

#varnish change
#<VirtualHost *:80>
<VirtualHost *:8080>


Apache client IP 127.0.0.1 problem

One important thing remains: All requests seen by Apache now look like they come from 127.0.0.1. This needs to be fixed in the logs obviously, so I can look at awstats when I have a hangover. But more importantly (or less, depending on the content Apache is serving) the other Apache modules (like mod_php) also see 127.0.0.1 as the client IP. If you are running an application that somehow uses $_SERVER['REMOTE_ADDR'] then the application might run into problems when all clients suddently have the same 127.0.0.1 remote address. If this is of no importance to you, like if you are only serving static HTML content, you can fix just the logging. On the other hand, fixing the problem completely is easier than just fixing the logging, although it does require the installation of an additional Apache module. Good thing we have ports to make that part easy.

To recap, you need to do one of two things:

  • Fix the Apache client IP problem with mod_rpaf2 (recommended)

or

  • Fix the Apache client IP logging only (not recommended)

Continue with the relevant section below.

Fixing the Apache client IP problem with mod_rpaf2 (recommended)

After I initially wrote this article, xi on #bsd-dk made me aware of the Apache module mod_rpaf, which can rewrite each HTTP request inside Apache, to make it look like it is coming from the X-Forwarded-For IP address. This is exactly what I need, it fixes logging and applications depending on the correct client IP all with one move. So I installed the port /usr/ports/www/mod_rpaf2/ and after removing the comment # from the LoadModule line the port added in httpd.conf, I added the following, also to httpd.conf:

[tykling@harmony /usr/ports/www/apache22]$ grep RPAF /usr/local/etc/apache22/httpd.conf
RPAFenable On
RPAFproxy_ips 127.0.0.1

And did an apachectl graceful after which everything is logged correctly, and all requests seen by PHP and other modules appear to come from the actual client IP. Neat.

Fixing the Apache client IP logging only (not recommended)

If you can't or won't install a new Apache module, there is still a way to make Apache log the correct IP. It involves defining a new logformat in the Apache config.

The Apache logging is currently set to something like this in my Apache configs:

CustomLog "|/usr/local/sbin/cronolog --symlink=/usr/local/www/logs/wiki.tyk.nu-access.log /usr/local/www/logs/wiki.tyk.nu-access-%Y%m%d.log" combined

Don't be alarmed by the fact that I am using cronolog to log with, a more normal logging statement without cronolog could look like:

CustomLog "/usr/local/www/logs/wiki.tyk.nu-access.log" combined

The combined logging format is (from the makers of Apache) defined in httpd.conf:

[tykling@harmony ~]$ grep combined /usr/local/etc/apache22/httpd.conf
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

I defined a new format called varnishcombined as suggested on the Varnish site:

[tykling@harmony ~]$ grep varnishcombined /usr/local/etc/apache22/httpd.conf
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" varnishcombined

This tells apache to log the X-Forwarded-For IP instead of the request IP which is 127.0.0.1 since Varnish is running on the localhost.

I then proceeded to change all my vhost CustomLog statements to use the new varnishcombined format instead of the standard combined.

Starting everything up

I restarted Apache and started Varnish in that order, and everything seems to work smoothly. Every request is logged in the Apache logs, with the correct IP address, which is exactly what I wanted. Varnish listens on port 80, receives the (entire, which is important in this context) request, and forwards it to 127.0.0.1 port 8080 where Apache is ready to hand out content. And best of all, I am no longer able to take down the server with slowloris.

--Tykling 18:54, 24 June 2009 (UTC)