In reality, ModSecurity takes a lot of time to implement well, especially if you have a large site. The core rules will almost certainly block legitimate user behavior–interrupting their business process with a generic error message.

Here are 8 tips that can help make your ModSecurity roll-out a success.

1. Configure the ModSecurity SecResponseBodyLimit to be at least as large as the largest text document served by the site. Even in log-only mode, this will block large response bodies!
2. Configure the ModSecurity SecRequestBodyInMemoryLimit to be at least as large as the size of php’s upload_max_filesize limit. Again, ModSecurity will block file uploads that exceed this value–even in log only mode.
3. Start in log-only mode. ModSecurity will tell you what it would have blocked, giving you an opportunity to add whitelist rules or otherwise tune you ruleset.
4. Whitelist load balancer health checks. This usually involves adding a whitelist rule for the load balancer’s IP.
5. Whitelist automation and APIs that requests HTTP documents. This usually involves either an IP, user-agent, or URL-based whitelist. These are often easy to miss amongst the torrent of alerts ModSecurity detects.
6. Consider disabling audit logging for invalid user agents, missing accept header, and unacceptable HTTP headers. This will significantly reduce the number of alerts that need to be analyzed, improving the chance of finding an alert that really matters.
7. Review apps that use WSIWIG editors to ensure they are validating and sanitizing user input properly. ModSecurity loves to block WSIWIG input, generating alerts ranging from SQL injection to XSS to system command injection. The way around this is to whitelist certain rules for these app URLs.
8. Create an operational plan to regularly review and act on ModSecurity alerts. Consider using ModSecurity Console to reduce the amount of time needed to analyze audit logs.

When you add an nginx reverse proxy layer on top of Apache, Apache thinks that all connections originate from the server running nginx. This creates a couple annoying problems:

• Every entry in the Apache access logs appears to come from the IP of the nginx server
• Securing sessions by checking that a user’s IP address hasn’t changed becomes more difficult.
  • This is especially true when using open source software. OS packages usually look for the client’s IP in the REMOTE_ADDR variable.

To resolve these issues, we’ll use the Apache mod_rpaf module to populate the REMOTE_ADDR using a special HTTP header inserted by nginx. A typical request would work as follows:

• 1.2.3.4 sends HTTP request to nginx server
• nginx determines that it needs to proxy pass the request to a back-end Apache server (e.g. by looking at the content-type or virtual host).
• nginx adds an HTTP header “X-Forwarded-For” with the client’s real IP
• nginx forwards (proxy_pass) the request to back-end Apache server
• mod_rpaf in Apache detects that the request is coming from the nginx IP, then substitutes REMOTE_ADDR with the contents of X-Forwarded-For
• Apache handles request as normal. Applications do not need to be aware of the reverse proxy.

To install mod_rpaf on the CentOS 5 box:

wget http://stderr.net/apache/rpaf/download/mod_rpaf-0.6.tar.gz
tar zxvf mod_rpaf-0.6.tar.gz
cd mod_rpaf-0.6

# Patch Makefile so it looks for 'apxs' instead of 'apxs2' (required
# when compiling under CentOS 5):
sed -ie 's/apxs2/apxs/' Makefile

make rpaf-2.0
make install-2.0

Create /etc/httpd/conf.d/rpaf.conf:

# Path to mod_rpaf-2.0.so, relative to /etc/httpd/
LoadModule rpaf_module modules/mod_rpaf-2.0.so

RPAFenable On
RPAFsethostname On

# Define our reverse proxy IP.  Only substitute client IP in
# when we receive a request from this IP.
RPAFproxy_ips 192.168.0.1

# The header where the real client IP address is stored.
RPAFheader X-Forwarded-For

Configure nginx to reverse proxy our CNAME IP address to the back-end container. We won’t go into installing nginx here, instead I’ve included the relevant configuration section in /etc/nginx/nginx.conf. This configuration says to reverse proxy all requests to the virtual host ‘myvirtualhost.com’:

server {
listen 80;
server_name myvirtualhost.com;

location / {
proxy_pass http://192.168.0.56;
proxy_redirect default;

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}

After restarting Apache & nginx, you should be able to successfully connect to your back-end Apache via the nginx reverse proxy layer. Inspecting the Apache environment will show a couple new headers, but other than that requests look the same as if clients were connecting directly without the proxy.

Start with a pre-created centos-5 OpenVZ template & install required packages:

# Create centos-5 OpenVZ container:
vzctl create 1056 --ostemplate centos-5-x86_64-minimal
vzctl set 1056 --ipadd 192.168.0.56 --nameserver 123.123.123.123 --save
vzctl start 1056

# Update software & install packages:
vzctl 1056 install yum
vzctl enter 1056
yum upgrade

# Install packages -- the ".x86_64" tells yum to only
# install the 64-bit packages and not the i386 packages.
yum install vim-enhanced.x86_64 mysql.x86_64 mysql-server.x86_64 \
httpd.x86_64 httpd-devel.x86_64 wget.x86_64 which.x86_64 \
php.x86_64 make.x86_64 gcc.x86_64 gcc-c++.x86_64

# Clean up leftover files from yum:
yum clean all

Next, tune Apache to fit our development 256MB OpenVZ container. If you have more memory dedicated to your container, consider increasing these settings. Edit the prefork MPM section of /etc/httpd/conf/httpd.conf:

<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;">&lt;</span>IfModule prefork.c<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;">&gt;</span>
StartServers       2
MinSpareServers    1
MaxSpareServers   8
ServerLimit      8
MaxClients       8
MaxRequestsPerChild  4000
<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;"><span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;"></span></span>&lt;/IfModul<span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;"><span class="__mozilla-findbar-search" style="padding: 0pt; background-color: white; color: black; display: inline; font-size: inherit;">e&gt;</span></span>

Let’s lock down the MySQL root user before starting up services:

mysql -u root mysql
mysql> update user set password=password('mynewsecurepassword') \
where user='root';
mysql> flush privileges;
mysql> exit

Start services and ensure that services start when the machine boots up:

chkconfig --levels 345 httpd
chkconfig --levels 345 mysqld
service httpd start
service mysqld start

Finally: test!

This will give you a slimmed down LAMP stack, suitable for running small web applications on top.

The Sunfire X4500 came with 48 500GB drives and 6 disk controllers.  We configured the array into 7 RAIDZ2 sets, then striped the 7 sets together, leaving 4 spares and two root disks.  Since we started with Solaris 5/08, we went with a ufs mirror for the two root disks (Solaris 10/08 lets you use ZFS root pool).  This gives us plenty of redundancy at the disk and controller level.

Next we set up several ZFS filesystems, essentially one filesystem for each website using NFS, and one ZFS volume for each iSCSI share.  For now, we are only using one iSCSI share — this will change soon.

The filesystems exported over NFS have no capacity limits, while the iSCSI volume was thin provisioned at 1TB.  We are currently using only using about 100GB of the iSCSI volume.  In the long run the thin provisioned volume will save us from having to resize the volume and extend the ext3 filesystem that sits on top.

The iSCSI volume was shared to an Redhat RHEL5 box running Open-iSCSI.  We found Open-iSCSI to generally work well until you have a network hiccup.  In our initial tests, a network delay of 3 seconds would cause the SCSI block device to disappear temporarily, freaking ext3 out and remounting read-only.

Our solution to network hiccups was to add a DM-Multipath layer between the iSCSI and ext3 layers on the RHEL box.  Multipath will sense network issues and queue I/O until the issue is resolved, or a configurable timeout expires.  The queuing behavior worked perfectly as we went about unpluging and pluging network cables on the RHEL box.  Side note: Multipath is a much more capable piece of software, allowing a machine to switch from one network path to another if it had redundant network paths to the SAN.

One gotcha to look out for when using a Solaris + RHEL5 + Multipath combo: a bug on the Solaris iSCSI target side causes the SCSI ID to be reported as much longer than Multipath can handle.  We got around the bug by extracting what appeared to be the unique SCSI ID portion.

Performance was near wire-spead for sequential I/O.  Random I/O was fantastic, thanks in large part to the 16GB of RAM the Sunfire was using as a giant disk cache.  The I/O performance was much better than the local SATA that the RHEL box came with.

With the iSCSI share created, shared, all appropriate iptables/ipf rules were set up, and the block device mounted, we then moved all of the OpenVZ containers to this RHEL box.  OpenVZ needed no special configuration — we simply mounted the iSCSI device on /vz using ext3.  All containers have been running smoothly since (except for a bad RAM module, but that was unrelated to iSCSI or OpenVZ).

The NFS shares have also been running smoothly, although we did run into a nasty Solaris ipf issue.  The problem is that after a few days, automountd on RHEL randomly hangs when attempting an additional mount.  Packet captures indicate that ipf is blocking either the inbound or outbound packets for the second TCP connection (we are not using UDP) to the ‘nfs’ port.  Disabling ipf immediately resolves the problem.  We are currently trying to reproduce the issue on a test server.

Even with the few issues that came up, we are still very happy with our new setup.  The Sunfire X4500, ZFS, and Solaris 10 matches many of the essential features of a more expensive “enterprise” solution.

http://www.solarisinternals.com/wiki/index.php/ZFS_Best_Practices_Guide

http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide

Before attempting to understand OpenVZ’s memory limits, let’s recap Linux’s memory setup. This background information is important to firmly grasp OpenVZ’s behavior. You can safely skip this section if you already understand Linux’s memory overcommit accounting.

Linux Overcommit Accounting

Memory in Linux refers to the combined RAM + swap values. Memory is allocated from this pool using one of the *alloc() functions. If malloc() is used to allocate memory, the pages are not zeroed out. malloc() is unique in one regard: the kernel can malloc() more memory than what is available in RAM + swap. This behavior of allocating more than the total available memory is called overcommitting memory.

Seems like a recipe for disaster, huh? Imagine… processes are humming along allocating memory and BAM! some process wants to use the n’th + 1 byte of memory and an out of memory (OOM) condition provokes the kernel to start killing processes.

Why does the kernel allow memory to be overcommitted? Because most applications do not actually consume all of the malloc()’d memory. If an app doesn’t use a memory location, it does not really need to subtract from your total available memory.

Consider a typical Apache process. A full 20-30% of an Apache processes malloc()’d memory may not be consumed. With memory overcommit, this slice of memory can be used by other applications. Java applications also allocate much more than they use, as do many scientific applications.

At the system-wide level, these “malloc’d(), but not consumed” slices of memory begin to add up. By overcommitting memory, these slices don’t take up physical memory locations — resulting in more efficient use of memory resources.

OpenVZ and memory overcommit

OpenVZ also has the ability to overcommit memory. When creating an OpenVZ container, we specify the vmguarpages and oomguarpages resource guarantees. vmguarpages represents the maximum guaranteed amount of malloc()’d memory a container may have, while oomguarpages represents the maximum consumed memory. These are called guarantees because (If your server has enough RAM + swap) container processes will never be killed due to OOM if they are within these limits.

What if your container exceeds the vmguarpages or oomguarpages value? This is where a new limit comes in: privvmpages. privvmpages represents the absolute upper limit on container memory. When you run free or `cat /proc/meminfo` in a container, you see this privvmpages value take form as “Total memory”.

The memory gap between privvmpages and the two resource guarantees (vmguarpages and oomguarpages) is not safe to use in an ongoing basis if the sum of all container privvmpages exceeds RAM + swap of the hardware node. This is because if the hardware node runs out of memory, it begins looking for containers using more than their guaranteed memory. The kernel kills a process from the container who’s physpages exceeds oomguarpages the most.

I’ve created a diagram that hopefully illustrates the relationship between these three key memory limits/guarantees. Also included in the diagram is kmemsize, which is used for non-swappable kernel memory–not very important as long as it scales with privvmpages.

How can we offer dev environments with low provisioning cost? This is one way:

• Set up OpenVZ on a RHEL5 server.
• Set up a web template (Apache, PHP5).
• Set up a MySQL template
• For each dev environment, deploy a web and MySQL container.

This 4 step process glosses over many important challenges. For instance:

• How do you access these machines over the network?
• How to get the latest code into each deployed web container?
• How to get the latest MySQL schema & associated dev data into the MySQL container?
• How to get code and schema changes out of the container and into another environment (e.g.: deploying to production)?
• How to upgrade/restart the VPS without having to coordinate with each developer?

I’m thinking we can overcome these challenges with some crafty configuration & scripts.

Network access

NAT all VPS servers for more fluid provisioning, less DNS updates. Web nodes occupy one network subnet, DB nodes occupy another. Use a database to track all VPS and build a simple webapp on top that provides bookmarks to each VPS along with CRUD ops.

Getting code into the VPS

Using our git repo, check out a working copy with the latest dev branch. Alternatively, NFS mount staff home directories in each VPS and work off that. I’d prefer the NFS approach, as it makes the VPS more transient: we can destroy and re-create the server using an upgraded template if desired without loosing code.

Getting DB schema into the VPS

Database migrations, similar to Rails (we use PHP for most apps). Moodle CMS also has a form of database migrations which happen via the web, which is a tad simpler. Doctrine ORM supports database migrations, although we use it only for new development and it won’t help our existing code. For older/legacy webapps, we’ll have to maintain a per-tool schema file to bootstrap our webapps.

Getting code and schema updates out of the VPS

Pull code changes using git into a central repo. Put all database migrations in the code, using Doctrine as described above, or schema bootstrap file.

Painless VPS reboots

If all database changes exist in code, and all code exists on the shared NFS server, then destroying & deploying a new VPS from an updated template should not impact a developer. Test data may disappear, which may cause concern.  Ideally, all important data would be saved in-code and easily imported into MySQL, through migrations or bootstrap SQL files.  Alternatively, we could place the MySQL data files on an iSCSI lun, at an additional provisioning cost.

Once we are able to quickly provision new dev environments, why not do the same for production apps? I think we could use the same code bootstrap & database migration code to deploy new application stacks on demand.