Lack of DNS redundancy takes down award-winning system

No matter how much cloud is used, if DNS is your single point of failure, you're going to have a bad time.

Orica, a multinational mining and chemical company based in Australia, has picked up the ITSM Project of the Year gong from the IT Service Management Forum Australia for the implementation of its project to consolidate its disparate local IT service experiences for users into a single cloud-based global solution.

While it is a good news story on how the cloud can improve company and IT processes, the project ran into a bit of a hitch yesterday, thanks to Optus. Yesterday, the telco suffered an a major DNS outage that took down its website, corporate VPN, and IP services.

Orica ended up being caught in the outage. As a company that has its ServiceNow incident management tool in the cloud, the DNS issue showed a singular point of weakness in the system.

"The tool was available, and the issue was that our external DNS was hosted by Optus and didn't work," Hubert van Dalen, global IT infrastructure and operations manager at Orica told TechRepublic.

"So all the email that ServiceNow would send out as updates on the major incident got stuck somewhere in the internet, and couldn't be resolved nor routed to the people that needed to read those emails.

"If you all do that through a single console, and that's gone, then your incident process becomes a lot less effective."

van Dalen said the outage had a ripple effect throughout the organisation, thanks to its highly automated process, and it is an issue that he says they are looking to solve.

"It became clear that we needed some other DNSes to have the external names resolved; internally, we didn't have a problem. Both of our DNSes were from Optus, so they were not split over two providers."

It's another great example of how an IT solution is only as strong as its weakest, in this case also non-redundant, link.

By Chris Duckett

Some would say that it is a long way from software engineering to journalism, others would correctly argue that it is a mere 10 metres according to the floor plan. During his first five years with CBS Interactive, Chris started his journalistic adven...