A little over a year ago, I wrote an article entitled "Digital Display Network Technology and Disaster Preparedness," in which I outlined some of the arrangements that WireSpring has made in order to withstand a disaster, and rebound after the coast has cleared (I mean that figuratively and literally - Florida is a hotbed of hurricane activity this time of year). In the wake of the devastation left by Hurricane Katrina, I thought it might be useful to document some of the upgrades that we've completed over the past year to make our organization even more resilient against a disaster, natural or otherwise. After all, even a thunderstorm or clerical error at your electric or phone company could leave you without power or connectivity for a while, so it's important to be prepared. Hopefully these tips will be helpful as you update your own disaster plans as well.
First off, everything that I outlined in my article last year - industrial server hardware, multiple datacenter facilities, distributed CRM, etc. -- is still in place and running nicely. Our distributed service infrastructure has proven to be extremely scalable and fault-tolerant, with all of our customers' digital signage and self-service kiosk networks running smoothly even during the worst of 2004's hurricane season. This experience also helped us identify ways to further strengthen our customer support tools in situations where our headquarters loses power or is otherwise uninhabitable.
Except for resources like online documentation and pre-recorded video tutorials, technical support must be provided by an actual human being. Thus, all of the server upgrades and off-site backup locations in the world can't help unless our support engineers are able to communicate with customers. Our first line of defense here is our live chat system, which many customers have adopted as their preferred support medium. Since it's a hosted service (just like our CRM suite), the live chat system continues to function in the event that our headquarters is shut down or left without power. Our support staff can log into the system remotely and have access to the same interface that they have in the main office. We've also beefed up our email setup from the original, redundant 2-tiered system to a multi-site redundant N-tiered system to ensure that email for the entire organization remains available.
Perhaps most importantly, we performed a total overhaul of our phone system, switching from a traditional analog PBX to a digital IP PBX. Aside from obvious benefits like ease-of-configuration and call queueing, the new system allows us to do something that I previously thought was impossible - provide seamless, off-site access to our phone system and inbound/outbound calls, even when our headquarters is without power or connectivity. Thus, if our facility loses power or we evacuate in anticipation of a disaster, each staff member - from sales to tech support to accounting - can use their physical handset or a software program to handle calls from an off-site location. Similarly, staff members who normally work out of other offices (domestic or international) can keep working without interruption. All inbound and outbound calls are automatically routed through backup PBX servers in other states, so callers never know the difference. The final technology component comes in the form of cellular modems, which are used for off-site connectivity in the event that cable or phone lines go down. (We're currently using Sprint's CDMA/1xRTT network, though we're really waiting for their EVDO service to become available in early 2006.)
Of course, all of this high tech stuff is for naught if you don't have a solid plan in place to actually make it work together, and on short notice. That includes getting our team to safety as quickly as possible, putting communications guidelines in place to make sure that staff members have a reliable method for accessing critical information, and running regular tests to make sure that all off-site failovers and backups work as expected. We had our first chance to test our updated systems when Katrina first slammed into Florida as a category 1 hurricane this August -- with less than 48 hours from initial hurricane watch to building evacuation. Even after our building was locked down and our staff dispersed, our secondary systems came up smoothly. All critical services, including email, phone, and live chat were automatically brought online, and our remote management system continued to tick along as if nothing had happened (which isn't surprising, since it's housed in several secure datacenters across the country). As a result, few if any customers noticed that we weren't in our headquarters, and conference calls, training sessions and support chats were handled just like any other day.
Obviously, we'd all like to live in a world free of hurricanes, earthquakes, tornadoes and worse. But it's unrealistic to ignore these threats or expect that they won't have a direct impact on us. Each step that you take to fortify your organization and prepare for these events helps you better thrive in the face of adversity and provide the support that customers have come to expect. And with more and more customers treating their kiosks and digital signs as critical business tools, this level of service is rapidly becoming a requirement in the marketplace.