Can you survive a 24 hour power outage in your office?
A little while ago, severe windstorms swept through the Northeast, felling trees throughout the region. Even a modern industrial park where we’re located wasn’t immune. Power was lost for more than 24 hours. Our office backup UPS systems failed when their batteries were exhausted. Office heat was suboptimal. Employees had to stay home.
Management’s task was to keep us running despite the challenges, honoring our commitment to servicing our customers responsively, .
Well, we made it work. We kept operations going with few negative effects. I’m sure our customers— many of whom expect industrial-strength disaster resilience and recovery from us—would have been happy if they had noticed.
So how did we do it?
- Like most parts of the world, natural events like powerful snowstorms, rainstorms, flooding, water main breaks, and so on can wreak havoc on your facility or the ability of your workers to get to work. Forethought is essential. Disaster contingency and recovery methods were discussed regularly and often and then plans were put in motion.
- Of course we purchased UPS systems to support our servers, storage systems, phone and alarm and entry systems. Their primary purpose was to allow for a gentle shut-down of our servers. Without a generator or massive UPS systems, it is difficult to keep servers going. We considered a generator but determined a hosting center was a better investment.
- We set up warning systems so we would be informed 24×7 of any issues relating to our infrastructure.
- In addition to a monitored alarm system, we setup video cameras so we could remotely surveil our space.
- We worked closely with our building managers so they were aware of our security and infrastructure needs.
- We set up a remote hosting installation in a certified facility that had industrial strength power, HVAC, flood prevention and security, as well as redundant network access.
- We backed up our products and other data in both our workspace and our hosting installation.
- We asked our employees to be prepared to work from home. Many already work some part of their week from home as is.
- We used bad weather for our drills with respect to working from home. Fortunately in New England, there are always opportunities. 😉
- We used SaaS services such as Github, Marketo, and SalesForce to allow universal access, as well as industrial-strength protection of our critical data that’s even independent of our hosting center.
- We switched to a VOIP phone system to give us in-the-cloud protection for telephone calls.
Ok, but what went wrong?
- Some of our employees didn’t get the message that power was down and came in to work. We need to improve our early warning system.
- Some employees didn’t have a duplicate work setup in their home. Measures are being taken to resolve that situation.
- Our power outage was unusual in that only one phase was out, potentially wreaking havoc on our HQ equipment that wasn’t protected by UPS. Quick thinking personnel shut everything off to prevent damage, but timing was fortuitous. That needs some further effort.
Ultimate result? Availability to customers was uninterrupted
- Ordinary business continued
- Some development and QA work was delayed
- Confidential documents and data remained encrypted and protected
- Disruption kept to a minimum
- All customer facing departments remained available.
Serving Fortune 2000 financial institutions, medical record processors, insurance companies, document management enterprises, shipping companies, and many more corporate and government enterprises depends on providing reliable, secure and modern products and services. In today’s corporate and governmental world, data security and 24×7 access to your critical vendors is non-negotiable. Suppliers, small and large, have to answer to these requirements. The products not only have to address customer needs, they also need to be reliable and secure; and the vendor has to be immediately available to the customer for emergency demands.
That’s our goal. I was very happy that our preparation and people’s fantastic efforts made for a good outcome. However, we won’t rest on our laurels.
Suggestions and comments?