ChMS - Virtualization

Yesterday we gave our Arena server a major, major upgrade. Not to Arena, per se, but to the underlying software. Up to this point (warning: technical details ahead!) we have been running Arena on Windows Server 2003 R2 and SQL Server 2005. Yesterday we made the move to Windows Server 2008 and SQL Server 2008. We also took this opportunity to virtualize the server using VMWare ESXi.

You may have thought of virtualization as a way to combine multiple servers onto one piece of hardware saving lots of money in the process on maintenance, licensing, and electricity. It does that, but for us that was only a small part of the appeal. We moved our Arena server onto VMWare ESXi (a free product!) yesterday because it offers us several advantages we don't otherwise have. We had already moved our testing server onto VMWare and found the benefits far outweighed the negatives.

Our first step was to shut down the server and pull all 6 1TB drives. We labeled them and set them aside just in case everything went wrong and we had to revert to the old Arena before the day was done (we did some of this out of order because we thought of it later, we should have copied this server to a virtual machine right away but we didn't think of it until after dinner...). The next thing we did was to install a fresh set of 6 1TB drives and configure them as a RAID 6 with one hot spare. RAID 6 gives you the ability to lose two drives without losing any data, at the cost of two drives worth of capacity. So in our case, that results in a 3TB RAID + 1 hot spare. We should be able to lose 3 of the 6 drives without losing any data.

The next step was to install VMWare. We made a mistake, in that we set the volume size as large as possible at 1.99TB (the expensive versions of VMWare can handle larger volumes, ESXi cannot, and we can't afford VMWare Enterprise Plus at this point). The problem was that after a few hours of work we realized the snapshots weren't working because we didn't leave room for the overhead. We kept getting a "file is too large" error. Highly annoying, but fixable. The path of least resistance was to start over and configure the volume with a lower capacity. Since it is easy to expand a volume but difficult to shrink it, and because a smaller volume is easier to work with if we have to move it to another machine, we went with 1TB. Considering our data is currently a little over 1GB we have plenty of room.

The biggest upside for us in using VMWare is the ability to take a snapshot of the hard drive. This way if we want to install something new and test it, we can take a quick "snapshot" and then if there are any issues we can easily revert to the prior version by restoring to the snapshot. The same is true before we run OS updates, SQL server updates, or install any new piece of software. There is another upside as well: We put the old drives back in our testing machine and then converted them to a virtual machine. This gave us an easy way to access the "old" arena server if we forgot something, can't remember how we set something, or need to copy something off.

Virtualization also allows us to create other environments as needed. For example, one Arena church gives each volunteer their own virtual machine running arena. That way the volunteer can do their coding and testing without disrupting anyone else. It's a great idea, and one that we might copy.

For now we have two matching Dell 2950's in use. Each is an eight core machine with 6TB of storage and 16GB of RAM. When you consider how powerful these things are they were amazingly cheap. We use VMWare on each of them. For now our main arena server runs only arena, and we have the "old" VM available if we need it. The testing server has our testing environment, a development environment, and also serves as a host for a bunch of low use servers (running linux) that we are saving money by virtualizing onto this box. This frees up room in our server room, saves us money on electricity, and also provides a huge speed boost for when those servers do get used.

There are a few downsides. There is a slight performance hit, although it is very slight (as fast as these boxes are, I don't think we will ever notice). The second one for us is that ESXi doesn't support more than four cores per virtual machine. For our primary box this means that we dropped from 8 cores to 4 when we virtualized. In watching our cpu usage we don't believe this will be a problem at this time, but it is highly annoying. Finally the 2TB limit is a bit bothersome, but at this point on these machines aren't anywhere close to that in what we are storing so we are fine.

We anticipate that by the end of the year we will move all of our non-mac servers (still don't have a good option for mac VM's) onto this hardware and remove another three servers or so from our server room. Although I don't know the exact cost savings, it will be significant in decreased energy for the machines, as well as for air conditioning etc.

We learned about the benefits of virtualization from Emil and David at Lake Avenue Church in Pasadena. They moved something like 31 servers onto two physical pieces of hardware. When we first met them they were preaching the amazing benefits and having done it, I see why.

VMWare ESXi provides a cheap and easy way to get into virtualization and start enjoying the benefits for your church right away. If you haven't made the move, it's time.

Joel