We service a lot of companies that have an in-house I.T. person. We are the “backup” when they can’t fix something or need extra hands. This is one that solidified our Outsourced only position at this company.
So around 2 PM today we all noticed something very strange. We were getting a lot of automated Offline tickets on the service board.
We called the in-house tech and asked what was going on, they informed us that No more remote sessions, nor emails were coming into the system at all. Lync/Skype showed nothing wrong until just about everyone got logged out at the same time and the calls suddenly started jumping off left and right. Everyone is complaining about their email. Suddenly the most vital of emails to ever exist in the history of the world plus infinity are not going through. The normal methods are not working… AND NOW OUTLOOK LOGGED ME OUT FIX IT NOW!!!!!!!!!
We rush down to the customers location and when we arrive on-site the fire alarm went off OF COURSE the fire alarm is going off now. What else could possibly be wrong.
The manager comes out and tells us all to exit out and for me to join him in the server room. One of the racks has smoke coming out the back of it. We pull the rack and see a full blown fire, no sparks, but a fire inside the primary exchange box. We pull it and then exit the room as fire dept comes in. The argon suppressors were going off, unbeknownst to us, so when we get outside me and the manager sound like our voices dropped 12 octaves. We had fun with that making a few firemen laugh. They didn’t want to chance it so they called an EMT in just to check us out. Argon is pretty safe but its better to be safe than sorry.
Firemen pull out the giant fans to blow the smoke out and then let us all back inside. I get inside and hear the ugly news.
Me: – The entire box is going to have to be replaced.
In House IT – So what does that mean?
Me – No email for the entire company for 24-48 hours.
I instruct the in house IT to change the voicemail recording and disconnect his phone.
In House IT- Thank you for calling our company support line. Unfortunately none of our agents can assist at the moment as a fire has destroyed the email server. All of your emails are safe and sound, however no incoming or outgoing email will go through for the next 24 – 48 hours. Thank you for your patience.
I politely excuse myself as I watch the in-house guy log out of the phone system, go into the break room and hooked up Netflix through a laptop to the TV. He started watching marvel TV shows on Netflix.
The EVP over IT came and asked what he was doing, saying his phone was getting blown up. We informed him of the issue and told him our fix will not be in place until the emergency shipping arrived tonight. (UPS Same day shipping is not cheap, but it is fast.)
EVP – So what do I tell the other VP,s Presidents, and Managers till then?
Me- Tell them there is nothing that can be done till UPS gets here so turn their phones off.
HE turned his phone off and sat down to watch netflix with the inhouse IT guy.
EVP – I am going to order pizza.
So what caused this fire you might ask? Well when I joined the manager in the server room, there still on the floor was the incriminating evidence. All wrapped up in the subway wrapper uneaten, save for a single bite out of it, was a foot-long subway meatball sub. In the back of the server box that caught fire? Marinara sauce and a half eaten meatball.
After completing the repair, i politely left a note on the door stating “No food or drink beyond this point” .
