Published by Pat Riehecky on 04 Aug 2009
A Summer’s worth of upgrades
It has been more than just “some time” since we last posted to our blog. Lots of things have happened. So, as a bit of information exchange here are some of the highlights.
Blade technology is now featured in the IWU datacenter. We have on hand a single HP C7000 blade enclosure with VMware ESX 4. It has only two blades at this time, but these two blades host a total of 26 running servers right now. I would expect we can get about 30 total servers virtualized between the two of them. That number may seem high to non-virtualization people, but it is actually a bit low. For redundancy we have reserved half of each blade for failures. If one blade should fail, we can run all the servers on the other. This gives us an extremely unsafe capacity of about 60 servers with just the two blades. As we add more blades to the enclosure, the safe number will increase much faster than linearly (but not exponentially). As we push more and more university servers into this environment we need to be more and more sure about the stability and reliability of the system. It is expected that an extra blade, redundant network card, and redundant fibre card will be added over the coming budget cycle. Here is a brief report on our current VMware cluster:
vCenter stats
VMware is not our only virtualization on site. Solaris 10 comes with a rather fascinating product called zones. I will be the first to tell you that I would rather run Linux (Debian if possible) than Solaris, but zones are a truly nifty tool. Zones are a very simple form of virtulization. For the UNIX savvy reader a zone is simply a bootable chroot, for the less geeky there are far better docs out there than I could emulate.
I bring up zones not just because they are somewhere on campus, but because of where they are. If you have checked your email or logged into my.iwu.edu since the outage last week, then you have used our new zones. That’s right, mail.iwu.edu and my.iwu.edu are virtualized.
What this means for students is simple. Remember the horror storries about registration and the big slowdown on my.iwu.edu durring that window? With only needing to buy one box to replace two our purchasing power went up rather drastically. Old my.iwu.edu had four processors at 1062Mhz (these are 64 bit SPARC microprocessors so don’t go thinking that your 3Ghz Intel/AMD is three times faster. The 3Ghz Intel/AMD chip is faster, but not by three times but wastes about twice the power of a SPARC) with 8192 Mb of RAM. Old mail.iwu.edu had two processors at 1593Mhz (64 bit SPARC again) with 4096 Mb of RAM. The system we purchased to replace both of them has a bit more power behind it. Weighing in with 32 processing cores at 1200Mhz (64-bit SPARC) and 16256 Mb of RAM this system should perform during registration. In non technical terms, each core can do one and only one thing at a time. It does that thing at the given speed. For well written software you gain about 1.7 things per core added. Registration gets slow because there are more things to do than cores to do them. With 32 cores availible, that should be a thing of the past. The my.iwu.edu zone has been given 12G of the 16 on the system for its own use along with the 85% of the core throughput under max load (if the system is not under max load then my.iwu.edu gets everything it asks for). For years there were complaints about the speed of registration, they were partially addressed with the new banner server last year. This server should address the remaining portion.
Staff members are not excluded from the benefits either. They still benefit from the speed improvements, but the primary value for them is a bit more social and communal. The old system consisted of two servers and two SANs. Each ran 24/7/365 and each one was covered under a very expensive support agreement. The new server has but one SAN attached to it. The new SAN is more power efficient and the new server is rated amongst the most ecologically friendly in the world. It cost about the same as the old servers did originally, but the current equipment was manufactured more efficiently and runs much more so. To top it off with the two systems running on one set of hardware, there are few idle systems in the datacenter. We are wasting much less money planning for capacity that only happens at registration. It just isn’t acceptable to crash under the load of registration, but more powerful servers draw more electricity. With them combined we can reduce the cost of idle time by reducing the idle time.The new setup saves the university money every second of every day and causes the creation of few greenhouse gases than their predecessors. All for an boost in speed and performance - not to mention reliability. Some staff members have expressed concern about the power draw and ecological effects of our computing resources on campus. We listened and are trying to reduce our carbon footprint without cutting any services. I cannot speak for how much we can reduce without cutting things, but do know your voice has been heard.
During that same email outage we also upgraded our authentication system to OpenLDAP. The performance boost has been overwhelming. My testing prior too deployment lead me to predict a 200% speed boost. Real world testing has been closer to 600%. What that means for you as users is simple. Any time you put your NetID and password there is a connection made to our LDAP server. The faster it responds, the faster you can get on with whatever you want to do. To make matters all the better, the LDAP system is now behind a load balancer (from Barracuda Networks). In the past IWU has maintained two LDAP servers and configured services to use both (one directly and one for failover). This was an acceptable solution. If one was down, then the other would be used. The problem is in time outs. Generally after waiting up to 180 seconds the second server would be tried. This three minute timeout often lead people to believe that a given service (my.iwu.edu usually) was not working when it was instead being very slow through no fault of its own. The load balancer takes care of the fail over all on its own. This means we can have a rather massive system failure without bothering you the end user. I think we can safely agree that is a great thing.
Speaking of failure, our previous LDAP system had a number of bugs that required great risk to patch. Now when I say great risk, what I mean is that there was a chance of losing all data within it when patching the server. Keeping that in mind, it should be somewhat obvious that we didn’t patch often and left several patches off because the risk was too great. The new servers are virtualized, so I can clone them whenever I desire, and OpenLDAP contains no such risk. I can do “online” patching in a full test environment without interrupting a single user. To put it into perspective, let us assume the patching risk of the old systems were true for updating your computer (this is false, please keep your computer up to date). If this assumption were true, every time you applied a system update there would be a 50% chance you lose every single piece of data on the computer. Generally when my friends have such serious bits of data loss they bring the computer to me, for the sake of the example and putting yourself in my shores, you cannot assume anyone will be willing to help you in any way. How often would you update it? Not often. This comes at a price. Some of the price is stability, some is functionality, and some is security.
And what of security? Your password was kept safe, and no one from off site had any good way of getting at it (unless you entered your password into a non SSL protected application/webpage), but there was more we could do. Originally, the internet was a safe place where people trusted each other and little was encrypted. That was some time ago, but many of the protocols that make online communication possible have lagged a bit. Our old LDAP server supported unencrypted connections. ITS as a whole made a commitment to avoid using them at all costs. We tried very hard and, I believe, avoided them in almost all ways. The option was still there however, and over the years some people have developed things that connected to our LDAP without asking for assistance. This created the possibility of risk. I don’t like that. Your password is important and I would much rather keep it secret and keep it safe. The new OpenLDAP servers have a simple security setting which denies all non encrypted connections. No matter how hard you try it will not listen for passwords over an unsafe connection.
Our last part of the LDAP upgrade was related to the wireless network. Anyone who used IWU-Wireless in the past should have noticed a 10-15 second delay in the connection. Any attempts since the upgrade should be almost instantaneous. This is partly the result of the increased performance of OpenLDAP, partly the result of the load balancer, and partly the result of an upgrade to our RADIUS enviornment. We are now on FreeRADIUS 2. There have been a number of serious gains in performance and persistance. There are some more upgrades planned for the wireless system, but I wont tip my hand too much on that account. We have some stuff in the works for the iPhone and iPodTouch along with other WiFi capibile smart phones. Gaming consoles are still a ways off (if they will support EAP-TTLS or PEAP we can talk deal, but….)