Network and Server Group

We have all seen the commercial for IBM where the technician comes into the data center to find that all of his servers have been stolen. Then it is pointed out to him by a co-worker that the data center, that was once overflowing with hardware, has been condensed down into one rack with an IBM blade server in it. While this is a bit of an exaggeration by IBM, it does convey their point quite eloquently.

There are a few hidden elements to this technology that IBM could not touch on in a 30 second promo window, so they touched on the main issue. This would be a “Cost of Ownership” issue. What they want you to believe is that you can do more with less hardware.

Containing the Monster

Over time, data centers have become an ever sprawling monster that gobbles up a university’s precious space resource. We see that here at IWU as well. What seemed to be an adequately “spec’d for space” data center has been eaten up by today’s demand for more and more services. It seems that every new service we provide requires at least one of it’s own dedicated servers (as demanded by the vendor) AND in some cases, the service is spread out across as many as 4 to 5 servers.

In the past, the answer was, CONSOLIDATE!!!! Put as many services on one piece of hardware as you can. This has been found to be a bad idea. Imagine, if you will, one server that has campus authentication (LDAP), DNS and DHCP running on it. Can the server handle this? Well, maybe, if it is adequately resourced enough. However, what happens when that server goes down? All those core services that keep the campus network connectivity up and running are all down. It doesn’t matter why all these services went down, the fact remains, all of our users can not access the internet.

The latest emergent technology that promises to help with this issue is Blade technology. Ooooh, Blade Servers, doesn’t that sound cool???? Well, don’t laugh, it is cool! Network vendors have been using this technology for years and now we see it popping up all over in the server area as well. What Blades actually do is allow your data center services to be consolidated onto one piece of hardware that acts and operates virtually like many different servers. So in essence, we are doing all our processing on less hardware.

Think about Blades this way; let’s say you have a Lamboghini Diablo and it has a whopping 530Hp, but you never take it up passed 65MPH (because that is the law). How much of that car’s resources go unused? Probably enough power is left unused to push 4 Yugos at 65MPH. Where does all that unused resource go? It remains unused and is lost. The same goes with servers. You have this incredibly spec’d out server, but it is only being pushed at half it’s capacity. We have paid for resources that are not being used. Wouldn’t it be nice if when a server’s resources were not being used, it could give that resource back so other services that may need it could take advantage of it? That is exactly what Blades do! It is a big community of resources that gets shared between all the virtual servers that are running on it.

Taming The Monster

Not only does today’s hardware practices turn our data centers into monsters that consume our space resources, but someone has to power all these pieces of hardware too. We have experienced this problem first hand in the past few months here at IWU. We are continuously trying to find ways to power all the servers without tripping the breakers and bringing down the existing servers. At one time, we plugged in a monitor that tripped a breaker and brought down Mail. But, due to our system administrators skill and cat-like reflexes, it was brought back up quickly and without much notice.

Currently in our data center we have about 7, 110V 20Amp circuits to power all our servers and network gear. 6 of these 7 circuits are at capacity. We have 1 circuit left to add new servers on. Granted, we are trying to remove some older servers too which will gives us a few amps back, but at best, this is a stop gap measure. We seriously need to think how we can use the power we have and use it in a more environmentally friendly way. This is also in compliance with IWU’s strategic plan. Well, here is a way we can make this happen.

The beast devours power. We need to tame it’s appetite a bit. We can do this by implementing Blade technologies in the data center. Blades, as I said above, make better use of resources by sharing all the resources between all the services. By doing this, power is not wasted on unused server resources as they have been in the past. “Unused resources” becomes a phrase that is no longer used because all our resources will be utilized efficiently. Not only does the university keep with it’s strategic plan by being “green”, it gets to save a bit of “green” as well.

A Necessary Evil?

The beast does serve a purpose. Gone are the days of pens, paper and abaci. Computers are here to stay. They will require power. They will take up space. They are necessary. We just need a way to control it, and in our attempts to control it, we need to make sure it stays reliable.

Currently, IWU has 2 LDAP servers. These are 2 independent servers that act as one. If one goes down, the other takes the load. It is a reliability concern. We need campus authentication to work, and it must work always. So, let’s analyze these 2 servers as our example. Both are powered up at all times. Both are spec’d out identically. The each take up rack space. Both sit idle for over 70% of the time. But, both are necessary!!! Redundancy can not be forsaken in a solution to make us more environmentally friendly or in our strides to do more with less. All aspects must coincide in our solution if our solution is to be fair and equitable.

Again, Blades address this issue. A Blade server using virtual servers allow us to create as many servers as we wish. If we run out of resources, we simply add another blade to the chasis to increase our resources. So, given this, we can create redundant servers to our hearts content. And believe it our not, the Sys Admins here at IWU have a fondness for creating redundant servers. They are weird that way.

Check Please!

So, who is picking up the tab on this. Hahaha Oh yes, the awkward moment at the end of the meal. Well, there is good news here too, believe it or not. Initially, for IWU to get rid of about 10 old servers, we would have to come up with about 15K. Hmmmm, what would 15 new individual servers cost? What would be the power consumption of 15 new individual servers? What would the data center footprint of 15 new servers be? This is how we should be thinking.

Think about the benefits here; less hardware, less power, less cooling, better redundancy, better resource utilization etc…, at about the same cost, if not cheaper. Like any new technology, there are pains with adapting a new way of thinking, but the pains are just birthing pains to a better way of computing; a more environmentally and responsible way of computing. A way that is all ready being adopted by our peers around the world.

Bottom Line

Illinois Wesleyan University is ready to embrace such a change. We see it in our strategic planning and in our people’s attitudes to become better and more responsible with technology. All we have left to do now is act upon what we know. We should be a leader and an example here to other universities and show them that IWU acts on what it believes in.

If you have been around IWU for a while you will have noticed how more and more systems are getting connected to the NetID system.  This is great for everyone.  From my end, there is only the one place where user accounts can be messed up (excluding the few systems which are in no way hooked up to LDAP, but that is hopefully coming…. hopefully).  From your end, there is just the one username and password to remember.  But is the project complete with just LDAP?

LDAP (Lightweight Directory Access Protocol) support is available on almost every piece of Enterprise grade software.  This is all well and good, but all that does is provide Singular Sign On.  One username and password that you must enter into everything.  This is close to a Single Sign On but also very far from the goal.  What then is Single Sign On?

Single Sign On means, if you visit Ames, login to the computer there, you should automatically have access to everything that you should have access to.  You should just be able to launch a web browser, point it at my.iwu.edu and automatically get access to your email - without typing a password.  For a real Single Sign On, you should only have to login once, not only have one username and password.

What am I doing about it?  I have just built a test system that will be able to connect a few services that were not able to be connected to LDAP.  This helps push forward the Singular Sign On.  It also supports GSSAPI, the foundational protocol for Kerberos v5.  Kerberos V5 is a secure way of doing Single Sign On.  In another post on this Blog I mention the Microsoft way of performing Single Sign On over SMB.  It is not secure.  Kerberos is very well secured.  In the end Kerberos is the only secure Single Sign On system we understand enough to implement.  As an added bonus this system will also speak SASL.  SASL is just a method of protecting passwords in transmission.  One of the supported SASL methods is GSSAPI.  The primary motivator for this project was SMB support.  SMB is the official name for Microsoft’s file sharing protocol.  Support for that was built in, and designed in such a way that it too can do GSSAPI.  Lastly, it would be a shame to lose the LDAP support that we have all grown to love.  So LDAP is included, but this LDAP server supports GSSAPI as well.  All together this gives, under a single username and password, LDAP, SMB, Kerberos, and SASL based logins, each of which is GSSAPI enabled.

What does this really mean?  When this comes on line we should be able to start fostering a GSSAPI friendly environment, this then can give us a secure Single Sign On and a Singular Sign On too.  There are still some loose ends, but the testing coming in a few weeks should tie most of those up.  When it is successful, there simply is not a major protocol we are likely to bring online here that cannot be supported from that single username and password.

Yeah, but what does this really mean?  It means that, if enough applications can be configured on campus to support GSSAPI, you will start to see a day where all the university servers just magically know who you are and grant you the access you deserve after a single login.  No more logging in to login so that you can login to a service.

Technologies used: OpenLDAP - 2.4.7,  Samba 3.0.24, Heimdal Kerberos 1.0, smbk5pwd 0.0.0

Is there a next step beyond this combination?

Yes, we can add to the mix a bit of software called Shibboleth.  I am still trying to get it figured out, but once that happens it should be transparently added one day, leaving no one the wiser.  Why Shibboleth?  It is an open standard, secure, and free.  Periodically I hear mention of using Google to host our email, the price is vastly lower than what it costs us to do it ourselves, but, if we go that way, we would need a secure way of transmitting who can have IWU access to Google resources.  This is where Shibboleth comes in.  Google likes it because it is an open standard, very secure, and easy to integrate large numbers of remote sites together.  They support other access methods, but none really compare to what Shibboleth really is.  Another factor that is driving Shibboleth into the main stream is Internet2.  Internet2 has standardized on Shibboleth as its authentication system of choice.  If IWU is going to get Internet2 (which is currently a no, but that cannot last forever), we will have to decide if we are going to allow Internet2 based logins.  It seems silly to say no, but if we say yes then Shibboleth is a must.

I haven’t explored all the capabilities of Shibboleth yet, but I honestly expect it to spawn the next big thing in authentication.  Shibboleth itself seems capable of so many things, but only covers so many possible access methods.  The successor to Shibboleth will drive Single Sign On and Singular Sign On into yet another uncharted realm.  That, however, is about 10 years out.  Why wait for the next big thing when such a great one is sitting here waiting only for the time it will take to understand it.

Realistically all important data should have backups.  This is all well and good, but backups get out of date, this is where RAID comes in.  RAID puts one or more disk into an active mirroring of the data so, in the event of a disk failure, no data is actually lost.  You can read more information on RAID at wikipedia.

There are really two types of RAID sets to consider in server environments  RAID 1+0 and RAID 5.  RAID 0+1 is never to be considered, if a single disk is lost the entire RAID set is in peril - regardless of how many disks are available.  RAID 1+0 uses the same number of disks for essentially identical performance, but without this problem.

RAID 5 is very fast for block level reads and relatively quick for block level writes.   It sufferers a mathematical bottleneck with the parity.  Every block has parity information on it.  This parity information will allow the block to be rebuilt in the event that the original data goes missing or bad.  Reads and writes automatically call on this parity bit for data integrity.  For non-real time operations RAID 5 offers the best performance, disk space, and protection combination.  With RAID 5 you get around 2/3 of the disk space inserted into the RAID set for data storage; the rest is consumed by the parity calculation.  RAID 5, however, is especially bad for databases.  Databases are not only real time, but also generally ignore block boundaries in writing data.  This can lead to one database commit having 10 or 12 disk writes, each of which must find, calculate parity, update the parity, and verify the parity write was successful in addition to just merely writing the data to disk.  While this may sound far fetched, I have personally seen one normal Oracle checkpoint create over a hundred scattered writes merely to rotate redo logs.

RAID 1+0 is much simpler.  Every disk has an identical copy at all times.  Each identical pair holds a percentage of every file so that the data is spread out more or less evenly.  If there were 4 identical pairs, each one would have about 25% of the data for a given file on it.  This makes reads and writes relatively fast.  The RAID set keeps track internally of what parts of what file is where.  This, however, comes at a rather high cost.  In RAID 1+0 you receive about 48% of the disk allocated to the set. Half of the disk becomes the on-line mirror and about 1% of the overall space is allocated for keeping track of what files are where.  The 1% usage is duplicated onto the second disk, so if two 100 G drives were added (200G) only about 98G would be available for usage by the RAID set.  Why would anyone want to use this instead of RAID 5?  RAID 1+0 is a real-time RAID set.  It does not have the problem RAID 5 has in verifying the data on either reads or writes.  The data is assumed accurate, that is assured by the extra disk allocated for duplication.  This skips over entirely the mathematical problems RAID 5 encounters on non-block writes.  Every database out there should only be run on RAID 1+0 and every best practice guide should say this.

There has been some mention that the new Banner server should be faster, the primary reason for this is the RAID switch.  Old Banner was on RAID 5 as advised by an off-site specialist.  The new Banner is on RAID 1+0, this one change should substantially improve the performance and, hopefully, lead to a more stable registration environment.

As I am sure that most know by now, IWU Information Technology is contending with capital budget requests to fund the replacement/upgrade of IWU’s network infrastructure. We are currently meeting with vendors who intend on proposing their solutions for this upgrade. The RFP not only deals with upgrading our LAN equipment to handle faster backbone and switching speeds out to the desktop, but has also made allotments for the vendors to bid on extending campus wireless, network access control solutions and network monitoring systems.

IWU Backbone and Switching Speeds

Currently IWU’s network operates on a 1Gig backbone with dedicated 100M out to the desktop. The backbone speed is the rate the data travels from the switch that is in your building’s data closet back to the core router in CNS. The dedicated 100M speed is the data transfer rate from the computer on your desk back to the switch that is in your building’s data closet. The proposed network gear will have the capacity to handle the next generation of network backbone speeds of 10Gig while giving the end user 1Gig data transfer speeds out to their desktop.

Look Mom, No Wires…

This Request For Proposal (RFP) also has the vendors bidding on a new wireless plan for the university. This plan is to encompass all buildings on campus with the potential to start adding wireless in outdoor common areas in the future. What we are hearing from many vendors is that the new wireless gear can handle commonly used ratified wireless speeds of 54Meg with the ability to upgrade to the faster, soon to be ratified, wireless speeds of 250-300Meg. I will not bore you with all the “techie” statistics, but I will tell you that these wireless speeds will be noticeably faster. These new controller based wireless solutions will allow users that are using the wireless network to roam around in their building without ever losing their wireless connection. And, if in the future the wireless is extended across the entire campus, you will be able to move about from building to building without ever dropping your wireless connection.

Get The NAC

We have all had to deal with, from time to time, a nasty worm or virus that has affected our computer. We have all had to deal with worms and virus on other peoples’ computers slowing the network down for us. In this day and age we have come to expect that in order for us to use the internet safely, some type of protection is needed. Then why is it that there are so many computers out there that are not sufficiently protected? I imagine that there are many reasons that will qualify as an answer to that question, but, it really makes no difference in resolving the problem. One of the fastest growing technologies that is being adopted, not only by universities world wide, but by the business world as well is; Network Access Control (NAC).

NAC is an appliance that sits on the network that makes sure that all users who access the university’s network resources have shown themselves to be “qualified”. The implementations for this technology are varied. This appliance can tie into the university’s campus authentication so we can ensure that those users that are on our network are legitimate IWU community users. This includes students, faculty, staff, alumni, retirees, vendors, guests etc… and all can have different levels of usage on our network. Not only does this appliance authenticate the user, but it also ensures that the user’s computer is sufficiently protected in order to join the network by checking to make sure that the computer is running an accepted revision of anti-virus software.

Think about this for a second. Wouldn’t it be nice if we could start taking a pro-active stance in keeping our network space free of those things that steal our resources (i.e. unauthorized users, virus, worms) while giving that same resource back to those that need it?

What about those computers that do not pass the NAC inspection? Well, that is the beauty of this appliance. If your computer is found not to be running the correct revision of anti-virus (or whatever the case may have been) the appliance can then direct you to a web page that allows you to down load and install the correct revision. Thereby allowing your computer to access the network.

Network Monitoring

The RFP also allots for a much needed network monitoring system. To have the ability to see into the network traffic is key for keeping network uptime service agreements. It is very difficult to head off a problem that can be devastating to a network if you have no insight that this problem was coming. Currently IWU Network Services Group (NSG) is using a few rudimentary tools to try to stay on top of network and server issues. It is becoming much more prevalent that NSG needs a way of staying on top of network issues as the LAN and WLAN usage is growing.

Some of the proposals for this are giving us the ability to set alarms for network usage and other “questionable” network behaviors. Some of these systems can tie in information from all the network equipment including the firewall and NAC. We would be able to see when the internet connection is having issues and ways to remediate these issues. We would also have that same ability to correct these same resource problems on IWU’s internal networks as well.

Time Line

After the vendors have submitted their RFP’s there will be a time of review to select those vendor teams that will best fit IWU’s plan for the network future.  If the network upgrade project is to be funded, then this project would be extended out over a 4 year 4 phase implementation.  I do think that this is really a better way to go since there are some growing issues that IWU would have to deal with.  Implementing a NAC solution and growing our wireless network should be something that we take seriously enough to implement with a lot of forethought.  I say this because there are going to be administrative issues that come with using these technologies.  We want to not only be sure that all the pieces fit together correctly giving IWU the best experience it can have with these new services, but also that we are able to maintain and grow these services seamlessly into the future.  To do this we have to consider personnel requirements and our ability to keep those personnel trained on this new gear.  We can not afford to keep growing IWU’s network infrastructure by adding service upon service without contemplating the amount of trained support personnel it takes to keep those technologies living up a service level agreement.

Bottom Line

The network that has served IWU so well in the past is getting tired. It is now about 7 years old and some of the critical core equipment is being retired by the vendors that produced it. We, IWU, have outgrown and are demanding more from the network then what it can deliver in it’s current state. It is time for IWU to put in place a network that will protect the user community, protect the investment that we have made in our network infrastructure and do all of this to further enable the university’s ability to achieve it’s strategic goals through the use of technology.

Suppose for a moment that I had a document that I wanted to share with you.  I could email to you, but what if it changed.  An endless string of emails could be sent, but isn’t that confusing?  What if you only need to see it occasionally?  You could pester me every time, but that hardly seems an improvement.  Suppose you only need to see it when I am out.  That is somewhat problematic.

This is the type of problem that can be solved with a file share.   Unfortunately here is where the problem gets far more complex.  Every major operating system has their preferred type of file share.  Windows is designed for SMB.  Mac OS is designed for AFP.  Unix is designed for NFS.  With some configuration and free software it is possible to make Mac OS and Unix speak SMB, but there aren’t any good free Windows drivers for either AFP or NFS (at least none that I can find).  With that in mind the current university file sharing system uses a software product call Samba to speak SMB.

Samba is free software, which places it firmly within our budget, and it is a tested enterprise grade solution.  Our implementation here, however, has not been as clean as I would like.  This mostly has to do with leftovers from the previous file sharing solution.  The previous solution was to build a Windows NT Domain and share the files over SMB inside of a domain container.

Domains are lock in technology.  Basically all that means is, the system has been designed to be easy to setup and hard to take down.  There are several insecure things that happen inside of a Domain based connection to a SMB share.  There are also a few secure things that go on.

First the secure things that SMB does in a domain.  It uses the system secret key (technically it is way more complex, but without a few hour lecture it isn’t worth explaining) to establish it is aloud to connect to the system the share is hosted on.  This prevents unauthorized systems from doing much more than annoying the file sharing system.  It can also utilize GSSAPI/SPNEGO based logins (for Active Directory sites only).  GSSAPI is basically Kerberos V5.  SPNEGO is a Microsoft protocol which does a lot of the same types of things.

SMB does some insecure things.  It is not end-to-end encrypted.  The login session controls access to the data stream, this is supposed to prevent hijacking.  The login connection is itself not secure, but it is encrypted.  The login session has non-dynamic encryption, the encrypted packets sent for login initially are the same packets that are sent two or even three years later if your password hasn’t changed.  To promote single sign-on, any share you connect to is initially handed enough data to try and authenticate you seamlessly.

Microsoft recognizes this problem and since Windows 2000 GSSAPI/SPNEGO has been the first try connection for any share.   Then it reverts to the old behavior.  When escaping from a Domain this is particularly problematic.  The GSSAPI/SPNEGO sent by the Domain member is encoded for the domain to use, a non-domain member will be unable to process these strong credentials.  The login will fail.  Then the system will revert to the old behavior, but these credentials are still not valid for the system.  The Domain member will try these methods each several times before allowing a username and password to be entered.   This timeout process can take up to a minute.  Non-domain members experience a similar problem.  Since this is SMB being spoken, and it was designed to be Domain driven vendor lock-in technology, the non-domain member system will try and determine the domain that the server belongs to for establishing something called “InterDomain Trust.”  Even though neither system belongs to a domain.  If both systems are workstations, the process fails almost instantly, but if one is a server, even a non-domain server to a non-domain client, the process must be attempted until it times out.  This takes about a minute, during which GSSAPI/SPNEGO and old style authentication are also attempted.

Some people have claimed that Windows Server does not have these problems, I am not sure why they have said this.  The protocol is the protocol, it doesn’t matter who implements it.  It is what it is.  And with the Windows Server solution we must pay a fee per user who connects to the system.  The expense is a bit high for no real gain.  Students often complain about a waste of tuition dollars, an identical free solution helps make these complaints a bit less justified.

What of AFP or NFS.  NFS is a security nightmare.  There simply isn’t a secure way to implement it.  People have tried, but ultimately it just cannot work, it wasn’t designed that way.  AFP is really only a Mac protocol.  While much of campus is Mac based, 80% or so is still on Windows systems.  AFP runs its security entirely through SSH, which is secure, but a very very high overhead for the service.  As such, by default, AFP is unsecured.  And adding SSH to the mix creates a whole separate security infrastructure which would need to be monitored in addition to AFP.  It would also generate a great deal more network traffic making the connection much slower.

I hear about Xythos occasionally.  While it looks curious, the main features it boasts of are less impressive when the high sticker price is compared to free alternatives.  Xythos does everything through WebDAV which is as secure as the web server providing the data.  It can be run through SSL which would provide strong end to end encryption.  Their is even a driver available to allow Windows clients to make this appear a normal share.

My concerns about it are rather many fold.  It uses a proprietary file system to store the user data.  How then am I to recover data from it?  The drive mapping is listed as a Windows feature, ignoring our 20% Mac population is just not acceptable for a campus wide solution.  Reports from the Internet call Xythos, “Not great 1 star.”  The big claim to fame is off-line editing, but this again is only a Windows feature.  How then is this data to be shared to the servers?  Long term we want to give all of you access to your server room storage space from an easy interface.  Right now WinSCP or FuGu are the only options for Windows/Mac systems.  It is assumed Linux users are happy enough with ssh.  If I cannot share the Xythos data to the webserver, your webpage is gone.  Nobody wants that.

Lastly, the beacon of hope is AFS.  Implementing AFS outside of a Kerberos Realm is pointless.  For the time being this halts this in its tracks.  OpenAFS is a free implementation of AFS, they have a free Windows driver as well.  Unix and MacOSX have had drivers for AFS for a long time.  AFS supports end-to-end encryption via Kerberos and off-line editing.  AFS also supports a kind of intelligent caching where writes occur locally and are then propagated up to the server in batch.  This makes AFS a possible candidate for large scale network file manipulation, but I would have to see some numbers to know for sure.

On the whole, AFS is a very exciting and appealing way to go, but without Kerberos it will never make it to IWU.  With that in mind we have selected Samba for SMB logins.  It is as secure as we can make it without spending any money.  There are some problems with it, but they are a part of the protocol.  That we cannot fix.  Kerberos is on my to-do list, but so are a hundred other things.

This entry was prompted by a discussion of starting the file sharing setup all over again.  Here is a log of thoughts and ideas on it as a hope that whomever is inconvenienced will at least see that we are trying to do the best we can to find a safe and easy solution for making file shares happen the way users expect them to.

While most of the finishing touches have been completed on the Power Project in the Data Center, another issue has surfaced. Power circuits in the data center are being overloaded and causing circuit breakers to trip as new gear is added. This may complicate and hinder the expedient deployment of new servers.

To date there have been 2 occasions where circuit breakers have been tripped by adding new gear. Each time this has resulted in entire racks of network and server equipment shutting down in the middle of peak network usage. Thankfully, all affected equipment came back on-line with no serious hardware or software related issues.

Currently NSG is working on a data center floor and electrical wiring plan to make the best use of the space alloted in the data center. This plan will include the placement of all current rack equipment, future rack equipment, electrical circuits and cable runs throughout the data center. The Plan will outline the following:

* Placement of the racks

* 110V 20Amp Circuit and receptical placement (Two circuits per rack)

* 220 3 phase Circuit and receptical placement

* Cable ladder and wire management systems

* Rack to circuit correlation

* Location of doors, tables, AC unit and other non-movable objects.

The purpose of this plan is to start an interaction with physical plant to provide secure and adequate power for the IWU data center well into the future and to eliminate the power concerns that face us today.

We will update as the power plan evolves…..

In a effort to keep the network gear performing at its fullest, January 18th Chris Rutledge and I will be getting here at 2am to redo the fiber optic patch cables. This process will include disconnecting the rest of campus from the network core. This means that, while the building will be accessible to itself, it will not be able to connect to anything else. The cables will be re-run in a way that makes it easy to identify which cable connects to which building, and it will also dramatically improve the air flow around our core switch. This should help keep it cool and increase its lifespan. If successful, this will conclude the data center cable re-organization.

There will not be a performance benefit for anyone else, except in the event of a patch cable failure, but it will protect the cables better in hopes of reducing the failure rate.

This  unfortunately has been put on hold while we obtain more labels  so that we can ensure all cables are clearly labeled as well as logically grouped.