Despite the IT consulting industry’s remarkable capacity to crush hope and turn its practitioners to pessimism, I remain an optimist. Viewing the proverbial glass as half full, I’m a heavy Twitter user, believing the social media service is an outstanding communication method and possesses much value beyond just marketing, brand-building, celebrity worship, or news-collecting purposes. A recent technical crisis proved me right.
Why do crises always strike on Friday afternoons?
Several Fridays ago the afternoon was proceeding smoothly. I was working through email and catching up on my To-Do list. A Twitter feed, meanwhile, updated itself on my desktop. After closing a ticket, I overheard another engineer describing a strange troubleshooting case: A client was experiencing difficulty connecting to the Internet. Everything had been fine. Nothing changed. Then, out of the blue, a client complained that WAN connectivity was intermittent.
I checked my ticket queue — it was clean, so I moved on to another task. I overheard two engineers troubleshooting a similar issue in our office for a second client. One, I realized, might be an anomaly, but two meant something was up. Of course… it was around 4:00 in the afternoon on a Friday.
Curiously, we noted we could connect via RDP to each affected client’s router; this meant the WAN circuits weren’t actually failing. Our engineers began suspecting an ISP was experiencing a potential DNS issue or routing problem.
While an engineer in my office explored the errors, I noticed on my Twitter feed another IT consultant was questioning whether anyone else was experiencing strange network issues. I immediately sent the technician a Tweet saying yes, we were. A hot-hitting exchange of Tweets commenced. Within 10 minutes, we were linked by telephone comparing notes.
Because of Twitter’s capacity to connect my office to another consultant combatting the same challenge, we were able to combine our information, eliminate other potential issues, and determine we were battling the same problem, granted at different client sites. Short follow-up Tweets and the use of a specific hashtag helped recruit a broader network of IT consultants struggling with the same failure.
With the assistance of other technology consultants, my office was able to quickly discover and confirm the issue was specific to SonicWall clients using multiple WAN circuits and automatic failover. Because we were using relevant hashtags and Tweeting aggressively, the diagnosis took but a matter of minutes, whereas hours or days my have been required had the same troubleshooting occurred using more traditional message boards and email.
Crowdsourcing a solution
Soon one of our engineers discovered a potential workaround. After testing, I posted the interim solution on Twitter. Others began verifying the fix within moments. And we let it ride.
Ultimately, a temporary failure in a SonicWall data center led to corruption in the way some routers confirmed WAN availability. A couple of hours later I even received a Tweet from SonicWall informing me the company had fixed its error. While a seemingly small detail, the additional information and vendor confirmation proved helpful when performing follow-up calls later with clients to explain what broke and why and review how we identified a workaround so quickly and confirmed the failure was repaired.
The episode taught me how helpful a social networking tool can be in a time of crisis. The network failure was a complex problem we were able to determine was widespread, and we were able to figure this out quickly, thanks to Twitter. Twitter even enabled us to get in touch with other consultants in real time, as we diagnosed the issue and identified and tested a temporary solution.
Note: This post originally published in TechRepublic’s IT Consultant blog in September 2012.