Your Database is toast get out the butter and jelly

It's Aloha Friday morning on my way to work and I get get a call from the Data Center that the mission critical system for the organization is acting strange. Users are unable to access the system. Moments latter I get a text page from the system that a process that needs to run is gone. I had scripts monitoring the system so I knew something bad happened. 

I logon on to the system and it's not normal and the process that needs to be running was missing. I spoke with the Data Center staff and determined that an operator error caused the problem. The database that was in production was an interim database version to migrate to a new permanent database later in the month. It had known bugs that we just stepped on by accident.

I call an emergency meeting with management to brief them that the database has been corrupted. I contacted the application vendor to take a look at the database integrity. I advised management to implement downtime procedures. Hours elapsed working to fix the database with no luck. The database cannot be repaired. My only option now is the last system backup. I asked the Data Center Manager to bring the backup tapes to me. I’m holding in my hands the tapes of the mission critical system of the organization. I need a successful restore to minimize the window of data loss.

I call another meeting to break the bad news to management and staff. It’s going to be a long night we need to inform our users that the system will be down for an extended time and we will be giving periodic updates on the recovery progress. Key personal needs to be on standby and available when called.

The restore process is initiated. Restoring a large system takes hours and patience waiting for it to complete. Once the restore completes we need to run the integrity checker utilities to make sure no corruption exists on the restore. The system was validated to release back to the users with some minimal data loss between the backup and the time of the database corruption. Made some minor tweaks on the system. It's back on-line for everyone to use.

I got in the office at 8am Friday and it’s now Saturday 8am. I just experienced a System Administrators nightmare. When you are coordinating a major Disaster Recovery effort it's all about teamwork. You need to remain calm as you become the point person that knows everything that's going on. All eyes are on you. My Project Management experience definitely came into play on this recovery effort that had to be put together on the fly. You really know what you are made of when you go through an experience like a major system outage. It's a real good feeling seeing everyone work together as a team.