The System Failure Assessment

It's a good idea to evaluate all your systems to determine on a scale of 0 - 5 what level of risk you are exposed too. Develop a spreadsheet to track all theses systems. Items for example to list on the spreadsheet are hardware, operating system, database, applications, vendors and any other information that you feel is important to support the system. While you are doing the assessment it's a good idea to update your Data Center floor plan diagram identifying where each system resides. Once you complete the assessment place a sticker on the hardware indicating that it was evaluated already and come up with a code to cross reference it back to your floor plan diagram.

Doing a system failure assessment is a big project that takes commitment. Depending on the size of your organization it could take months to complete depending on how deep you dig for information. I suggest you do a thorough examination that will expose all vulnerabilities. While performing the evaluation it's a good time to review all your support contracts. You also get a heads up to budget for system upgrades or replacements. You really find out a lot of information to present to management from a system failure assessment.

Grading your findings for example could be a 5 for the highest level of risk if it's a black box that just runs and no one really has a clue how to support it because the person who installed it left 10 years ago and it just keeps on running, don't re-boot it ever. If you are running on an old hardware platform and it's off support and you are unable to patch the operating system that will be a level 5 risk. Just a few examples to give some ideas on how to rank your risks.

The System Failure Assessment will need to be a living document updated periodically to keep it effective and valuable. It will serves as a disaster avoidance / disaster recovery document. It's your decision how to act on level 3 - 5 risks. Don't wait act now.

The Experience Bank

Once you learned a skill or gained experience it's with you for life. You continue to develop it or put it aside in the most most precious storage your brain. Most people will continue to learn new skills and gain new experiences.

Retrieving skills and experience from deep storage in your brain does work it might be a little rusty and in need of some polishing up. It will comeback to you when you are in the situation to use it. I'll give an example of riding a bike. You may have not ridden a bike for 15+ years and when you get on the bike again it will all come back to you.

Your Database is toast get out the butter and jelly

It's Aloha Friday morning on my way to work and I get get a call from the Data Center that the mission critical system for the organization is acting strange. Users are unable to access the system. Moments latter I get a text page from the system that a process that needs to run is gone. I had scripts monitoring the system so I knew something bad happened. 

I logon on to the system and it's not normal and the process that needs to be running was missing. I spoke with the Data Center staff and determined that an operator error caused the problem. The database that was in production was an interim database version to migrate to a new permanent database later in the month. It had known bugs that we just stepped on by accident.

I call an emergency meeting with management to brief them that the database has been corrupted. I contacted the application vendor to take a look at the database integrity. I advised management to implement downtime procedures. Hours elapsed working to fix the database with no luck. The database cannot be repaired. My only option now is the last system backup. I asked the Data Center Manager to bring the backup tapes to me. I’m holding in my hands the tapes of the mission critical system of the organization. I need a successful restore to minimize the window of data loss.

I call another meeting to break the bad news to management and staff. It’s going to be a long night we need to inform our users that the system will be down for an extended time and we will be giving periodic updates on the recovery progress. Key personal needs to be on standby and available when called.

The restore process is initiated. Restoring a large system takes hours and patience waiting for it to complete. Once the restore completes we need to run the integrity checker utilities to make sure no corruption exists on the restore. The system was validated to release back to the users with some minimal data loss between the backup and the time of the database corruption. Made some minor tweaks on the system. It's back on-line for everyone to use.

I got in the office at 8am Friday and it’s now Saturday 8am. I just experienced a System Administrators nightmare. When you are coordinating a major Disaster Recovery effort it's all about teamwork. You need to remain calm as you become the point person that knows everything that's going on. All eyes are on you. My Project Management experience definitely came into play on this recovery effort that had to be put together on the fly. You really know what you are made of when you go through an experience like a major system outage. It's a real good feeling seeing everyone work together as a team.