maxgarrick.com

A short while ago we rolled out ModSecurity on three of our Apache web servers. The benefits of ModSecurity are clear: protection against most blatant web-based attacks, like SQL Injection and Cross-site scripting. It also acts as a last line of defense against information leakage, like PHP errors and directory listings.

In reality, ModSecurity takes a lot of time to implement well, especially if you have a large site. The core rules will almost certainly block legitimate user behavior–interrupting their business process with a generic error message.

Here are 8 tips that can help make your ModSecurity roll-out a success.

1. Configure the ModSecurity SecResponseBodyLimit to be at least as large as the largest text document served by the site. Even in log-only mode, this will block large response bodies!
2. Configure the ModSecurity SecRequestBodyInMemoryLimit to be at least as large as the size of php’s upload_max_filesize limit. Again, ModSecurity will block file uploads that exceed this value–even in log only mode.
3. Start in log-only mode. ModSecurity will tell you what it would have blocked, giving you an opportunity to add whitelist rules or otherwise tune you ruleset.
4. Whitelist load balancer health checks. This usually involves adding a whitelist rule for the load balancer’s IP.
5. Whitelist automation and APIs that requests HTTP documents. This usually involves either an IP, user-agent, or URL-based whitelist. These are often easy to miss amongst the torrent of alerts ModSecurity detects.
6. Consider disabling audit logging for invalid user agents, missing accept header, and unacceptable HTTP headers. This will significantly reduce the number of alerts that need to be analyzed, improving the chance of finding an alert that really matters.
7. Review apps that use WSIWIG editors to ensure they are validating and sanitizing user input properly. ModSecurity loves to block WSIWIG input, generating alerts ranging from SQL injection to XSS to system command injection. The way around this is to whitelist certain rules for these app URLs.
8. Create an operational plan to regularly review and act on ModSecurity alerts. Consider using ModSecurity Console to reduce the amount of time needed to analyze audit logs.

In this post I hope to show how to configure nginx as a reverse proxy to a back-end CentOS 5 server running Apache.

When you add an nginx reverse proxy layer on top of Apache, Apache thinks that all connections originate from the server running nginx. This creates a couple annoying problems:

• Every entry in the Apache access logs appears to come from the IP of the nginx server
• Securing sessions by checking that a user’s IP address hasn’t changed becomes more difficult.
  • This is especially true when using open source software. OS packages usually look for the client’s IP in the REMOTE_ADDR variable.

To resolve these issues, we’ll use the Apache mod_rpaf module to populate the REMOTE_ADDR using a special HTTP header inserted by nginx. A typical request would work as follows:

• 1.2.3.4 sends HTTP request to nginx server
• nginx determines that it needs to proxy pass the request to a back-end Apache server (e.g. by looking at the content-type or virtual host).
• nginx adds an HTTP header “X-Forwarded-For” with the client’s real IP
• nginx forwards (proxy_pass) the request to back-end Apache server
• mod_rpaf in Apache detects that the request is coming from the nginx IP, then substitutes REMOTE_ADDR with the contents of X-Forwarded-For
• Apache handles request as normal. Applications do not need to be aware of the reverse proxy.

To install mod_rpaf on the CentOS 5 box:

wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
tar zxvf mod_rpaf-0.6.tar.gz
cd mod_rpaf-0.6

# Patch Makefile so it looks for 'apxs' instead of 'apxs2' (required
# when compiling under CentOS 5):
sed -ie 's/apxs2/apxs/' Makefile

make rpaf-2.0
make install-2.0

Create /etc/httpd/conf.d/rpaf.conf:

# Path to mod_rpaf-2.0.so, relative to /etc/httpd/
LoadModule rpaf_module modules/mod_rpaf-2.0.so

RPAFenable On
RPAFsethostname On

# Define our reverse proxy IP.  Only substitute client IP in
# when we receive a request from this IP.
RPAFproxy_ips 192.168.0.1

# The header where the real client IP address is stored.
RPAFheader X-Forwarded-For

Configure nginx to reverse proxy our CNAME IP address to the back-end container. We won’t go into installing nginx here, instead I’ve included the relevant configuration section in /etc/nginx/nginx.conf. This configuration says to reverse proxy all requests to the virtual host ‘myvirtualhost.com’:

server {
listen 80;
server_name myvirtualhost.com;

location / {
proxy_pass http://192.168.0.56;
proxy_redirect default;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

After restarting Apache & nginx, you should be able to successfully connect to your back-end Apache via the nginx reverse proxy layer. Inspecting the Apache environment will show a couple new headers, but other than that requests look the same as if clients were connecting directly without the proxy.

Creating a minimal LAMP stack in OpenVZ:

Start with a pre-created centos-5 OpenVZ template & install required packages:

# Create centos-5 OpenVZ container:
vzctl create 1056 --ostemplate centos-5-x86_64-minimal
vzctl set 1056 --ipadd 192.168.0.56 --nameserver 123.123.123.123 --save
vzctl start 1056

# Update software & install packages:
vzctl 1056 install yum
vzctl enter 1056
yum upgrade

# Install packages -- the ".x86_64" tells yum to only
# install the 64-bit packages and not the i386 packages.
yum install vim-enhanced.x86_64 mysql.x86_64 mysql-server.x86_64 \
httpd.x86_64 httpd-devel.x86_64 wget.x86_64 which.x86_64 \
php.x86_64 make.x86_64 gcc.x86_64 gcc-c++.x86_64

# Clean up leftover files from yum:
yum clean all

Next, tune Apache to fit our development 256MB OpenVZ container. If you have more memory dedicated to your container, consider increasing these settings. Edit the prefork MPM section of /etc/httpd/conf/httpd.conf:

<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;">&lt;</span>IfModule prefork.c<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;">&gt;</span>
StartServers       2
MinSpareServers    1
MaxSpareServers   8
ServerLimit      8
MaxClients       8
MaxRequestsPerChild  4000
<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;"><span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;"></span></span>&lt;/IfModul<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;"><span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;">e&gt;</span></span>

Let’s lock down the MySQL root user before starting up services:

mysql -u root mysql
mysql> update user set password=password('mynewsecurepassword') \
where user='root';
mysql> flush privileges;
mysql> exit

Start services and ensure that services start when the machine boots up:

chkconfig --levels 345 httpd
chkconfig --levels 345 mysqld
service httpd start
service mysqld start

Finally: test!

This will give you a slimmed down LAMP stack, suitable for running small web applications on top.