CXO

Firewall install becomes a lesson on how to work with a contractor

An IT manager expected a few snags during an installation of a firewall system into his Ethernet network. Check out how the manager met the challenges that threatened to jeopardize the project.


If you are using a third-party contractor to implement an IT system, here are strategies to help you manage that relationship and move down the road to project success.

Our IT team recently worked with an outside contractor to install a Cisco PIX 535 firewall system into our Ethernet network. Although the project was completed on time, we were forced to jump several hurdles to achieve success. The problems we faced included ambiguity that resulted from sharing project responsibilities between my staff and the contract workers. Here are some of the lessons we learned that could help ensure the success of your next project.



Address assumptions
We planned a collaborative project (our IT team and contractor would work together on the implementation) to ensure that we shared knowledge and experience in working with the PIX firewall system. Since our team was going to administer the system, we decided that the best way to learn more about it was to be closely involved in the setup and installation of the firewall. Although this plan had benefits, it also had a disadvantage; there was an ambiguity during implementation about who was ultimately responsible for each project task.

The mistake was assigning two people, one from the contractor’s team and one from our team, to each task. This strategy helped us meet our goal of fostering collaboration. But major assumptions held by both sides prevented a smooth installation.

For example, our system administrator missed a key interaction between our Web and mail servers that caused a critical autoresponse function to fail. She thought the contractor’s team would have handled all interactions within the DMZ. (A computer network DMZ, as defined by WhatIs.com, is a computer host or small network inserted as a "neutral zone" between a company's private network and the outside public network. A DMZ prevents outside users from getting direct access to company data on a server.)

Meanwhile, the contractor had assumed that the ultimate responsibility and follow-through belonged to our IT team. After all, it was our network. On the other hand, we assumed that it was their responsibility because we were paying them.

Managing team expectations and assumptions and clearly defining roles and responsibilities are critical to a project’s success. We should have discussed these assumptions and expectations during our initial planning sessions.

Plan together
During the planning meetings, the contractor designated the project managers and technical team they would provide. But it was up to us to ask the tough questions and ensure that the system would integrate successfully. The trouble was we didn’t know all the key questions to ask.

We knew enough to ask how the firewall would handle traffic to and from the Internet. Traffic from a higher security level to a lower security level was allowed by default, so we configured our interfaces accordingly. However, we did not know what effectively happens when an access list is applied to an interface. We later learned that all traffic to any security level is compared against the access list. This explained why all our DMZ machines were unable to browse the Internet except our Web server, which had port 80 opened for HTTP traffic.

The contractor’s team was always taking copious notes as our team excitedly discussed all the apparent dependencies. However, we neglected to account for interactions between our Web and e-mail servers and for allowing a legacy application access to the DMZ.

We should have asked the contractor to draw from their expertise and experience to help us address the hidden issues and ease the strain of transition. Setting up another meeting, where the contractor’s team did most of the talking, would have helped us create a more thorough plan.

Installation day relief
On the day of the installation, we had to contact our ISP in order to reset the ARP tables on our router. In the past, we had difficulty making technical support calls to the ISP, so this time, we e-mailed reminders and made phone calls the week before, the day before, and the morning of the installation. To the credit of Murphy’s Law, the router was never reset at the requested time. The installation team spent 45 minutes waiting on hold for an off-hours engineer. You can imagine the stress and anxiety this caused.

We should have dedicated a contact person on our team to deal with the ISP. We could have contacted the ISP shortly before the installation to ensure we got an engineer online. Our contact person, who would not have to be technically savvy on the issues, could have expedited the process by contacting the ISP engineer and allowing the installation team to focus on the task at hand.

Escalate to overcome another obstacle
After the installation of the PIX 535, our users had difficulties communicating via FTP to our DMZ machines. Although HTTP traffic was uninhibited, FTP and telnet sessions would time out. Both teams tried everything, from applying patches to our Web servers and changing NIC card configurations, to altering DNS entries in order to accommodate reverse DNS lookups. Although each change provided limited help, the real solution eluded us. Two weeks had passed since the day of installation, and we still had no answer.

I finally sat down with our account representatives and honestly described our frustration at the lack of progress on this outstanding issue. They began to pool in resources from within their company. In addition, they found several solutions from Cisco to try. It turned out we just needed a 10/100 switch on our DMZ network. (We had been using an existing hub to save money.) Had we addressed this issue earlier, we would have considerably shortened our wait time.

When we encountered another problem with SMTP mail communications, we had learned our lesson. We quickly escalated the problem and got our answer within hours. The solution was to disable a feature in the firewall configuration enabled by default.

Lessons learned
Here is what I would recommend for this project, or similar projects, when working with contractors:
  • Plan better: Don’t make assumptions about what role everyone will be playing. During planning meetings, state even the most obvious details—such as who is ultimately responsible for each task.
  • Take advantage of the contractor’s experience: Plan a meeting where they do most of the talking. Ask them about what problems they have experienced in the past during similar projects.
  • Proactively plan prior to installation day: Consider worst-case scenarios and ways to prevent them prior to installation day.
  • Escalate sooner rather than later: IT teams pride themselves on their self-sufficiency in finding solutions. But this approach should include escalating issues to find solutions more quickly.

Finally, I believe that candid discussions with contractor representatives will help you complete a successful project.

The result of the problems we experienced? The project took two or three weeks longer than it should have. Luckily, the contractor charged a flat fee, so the additional time did not increase the amount we paid them. We certainly learned a great deal from our PIX firewall installation—not only in the technical details but also in contractor and project management.

What would you add to the list?
Have you trusted an outside firm to complete an important project in your shop? What advice would you add to Chen’s list? Post a comment or send us a letter.

 

Editor's Picks

Free Newsletters, In your Inbox