Discussions

Internet goes down across an organization. What do you do to troubleshoot?

Tags:
+
0 Votes
Locked

Internet goes down across an organization. What do you do to troubleshoot?

FredTill
You are an IT professional working for a large organization. The internet goes down across the organization and you need to act quickly. What do you do to remain focused and stay calm? What steps would you take to trouble shoot the issue?
  • +
    0 Votes
    muhammadsalmankhan

    Have an SLA in place with your ISP.
    Keep more than one or ISPs (Backups)
    First assess your own network, routing, etc.

    +
    4 Votes
    robo_dev

    Then get to work looking into the issue....

    +
    2 Votes
    jqbecker

    If so, shame on you.

    +
    0 Votes

    My thoughts as well! Anyway, the first place to check is not with your hardware or systems, BUT instead contact your service provider!

    +
    0 Votes
    Slayer_

    Then work backwards

    +
    0 Votes
    OH Smeg

    NA Slayer that's wrong.

    You first check the ISP then Modem and if they are OK work back from there.

    I remember a time when the company insisted that the ISP was working and I wasted hours only to finally be told that the ISP was down due to an digging incident where a end loader had knocked over a Power Pole and torn the Fiber Optic Cable and it was in the process of being repaired.

    Col

    +
    0 Votes
    lucvdv

    Run a traceroute to an external IP -- by numeric IP, not by name (and preferably one you know will reply, for example 8.8.8.8).
    If that just shows you that your packets get through all the way to beyond your ISP, find out why DNS stopped working.

    +
    0 Votes
    Ryalsbane

    Check the modem?!!! Seriously? If you are a large organization you won't be on a "modem," even a DSL broadband modem. If you are, then that is the problem. You need a complete redesign of your network from the ground up and upgrade of your infrastructure.

    +
    0 Votes
    Slayer_

    I don't know what you call that box that the fibre goes into...
    In our server room its about 6 by 6 by 2, has flashing lights, looks sort of like a modem.

    +
    0 Votes
    Slayer_

    I'm a computer programmer, I don't have to know how it works, just how to break it

    +
    0 Votes
    OH Smeg

    Well I call that box in rack in the server room with flashing LES's that gives you an outside connection to the world a Modem.

    The ISP here calls that Box a Cable Modem meaning it has Fiber Optical Cable coming into it and is the connection between here and the Internet.

    So Ryalsbane what do you call that box with the LED's on the front of it?

    Unless of course you mean we should all be using a 3/4G connection to the outside world but that's not going to happen here at least because it's way too expensive and it's still called a Modem even though it has a Router in it. :^0

    Col

    +
    0 Votes
    Slayer_

    Glad to hear I'm not totally wrong :)

    +
    0 Votes
    OH Smeg

    Even the Chip Set or Integrated Circuits used in these devices are called Modem Chips or things like Modem on/in a Chip.

    Col

    +
    0 Votes
    a.portman

    Verify the Internet is down "everywhere". Then start at the connection and work in. Otherwise, start at the desktop and work out.

    +
    0 Votes
    mjd420nova

    Using just a single laptop, connect to the modem to check access from there. If that works , it's the network/server if not access there, call your ISP.

    +
    1 Votes
    jtuck004

    the boss's nephew, who tries to download the entire Internet onto his Chromebook and complains that "the Internet" is down 4x a week. And while you are in your chair, take a glance out the window and make sure there isn't a backhoe working next to the service entrace for your T1.

    With that out of the way, take your laptop and, listening for the sounds of grumbling from other users as you make your way to the main router, (so as to make sure it is really down for everyone). Then, by looking to make sure a light is on at the module where your T1 plugs in to the router, and thus making sure they haven't tripped the breaker, again, by plugging the coffee pot into the same receptacle, again, see if you have service to the switches by plugging in your laptop (into which you have hardoded an IP address and DNS server) to see if you get access at the router. You do have a straight-thru cable with you, right?

    Oh yeah, and if you find that Internet access is being provided by a modem, you aren't in as large an organization as you think.

    +
    0 Votes
    mamccfts

    I would... Keep Calm and Chive On

    +
    0 Votes
    aj42617

    *Check if the issue is 1 device or network wide?
    If network-wide this is an issue then reboot your router and/or Firewall, wait 5 minutes and check for connectivity.
    *Then contact your ISP for confirmation of connectivity.

    +
    0 Votes
    Ryalsbane

    ...would continue to function. I would remain calm and focused due to having planned for this contingency when I designed (or redesigned when I took over for the previously ousted Network Manager/Admin) and implemented the network infrastructure to include redundant failover paths to the internet. My customers (company's employees and management) would not be aware of a failure until I announced it. I would know of the failure from the 24/7/365 monitoring and notification subsystems in place for this purpose.

    Assuming a basic go/no go monitoring with notification subsystem and that the local network is still functioning, I would do as lucvdv@ suggested. To get a quick, assessment of the scope of the failure and where the potential failure point is I would perform a traceroute to multiple internet destinations. The results will quickly tell me if the problem is with my local premises equipment, between the LAN and the ISP or with the ISP premises equipment. Once the scope of the problem is determined, if the problem is with local premises equipment I follow a reiterative "divide and conquer" process using traceroute, ping, netstat, nslookup, etc., to test for DNS and routing issues ultimately continuing through until the problem is fully isolated.

    +
    0 Votes
    OH Smeg

    But as you are not using a Modem you obviously don't have a connection to the WWW. So it doesn't really matter to you does it? :^0

    Col

  • +
    4 Votes
    robo_dev

    Then get to work looking into the issue....

    +
    2 Votes
    jqbecker

    If so, shame on you.

    +
    0 Votes

    My thoughts as well! Anyway, the first place to check is not with your hardware or systems, BUT instead contact your service provider!

    +
    0 Votes
    Slayer_

    Then work backwards

    +
    0 Votes
    OH Smeg

    NA Slayer that's wrong.

    You first check the ISP then Modem and if they are OK work back from there.

    I remember a time when the company insisted that the ISP was working and I wasted hours only to finally be told that the ISP was down due to an digging incident where a end loader had knocked over a Power Pole and torn the Fiber Optic Cable and it was in the process of being repaired.

    Col

    +
    0 Votes
    lucvdv

    Run a traceroute to an external IP -- by numeric IP, not by name (and preferably one you know will reply, for example 8.8.8.8).
    If that just shows you that your packets get through all the way to beyond your ISP, find out why DNS stopped working.

    +
    0 Votes
    Ryalsbane

    Check the modem?!!! Seriously? If you are a large organization you won't be on a "modem," even a DSL broadband modem. If you are, then that is the problem. You need a complete redesign of your network from the ground up and upgrade of your infrastructure.

    +
    0 Votes
    Slayer_

    I don't know what you call that box that the fibre goes into...
    In our server room its about 6 by 6 by 2, has flashing lights, looks sort of like a modem.

    +
    0 Votes
    Slayer_

    I'm a computer programmer, I don't have to know how it works, just how to break it

    +
    0 Votes
    OH Smeg

    Well I call that box in rack in the server room with flashing LES's that gives you an outside connection to the world a Modem.

    The ISP here calls that Box a Cable Modem meaning it has Fiber Optical Cable coming into it and is the connection between here and the Internet.

    So Ryalsbane what do you call that box with the LED's on the front of it?

    Unless of course you mean we should all be using a 3/4G connection to the outside world but that's not going to happen here at least because it's way too expensive and it's still called a Modem even though it has a Router in it. :^0

    Col

    +
    0 Votes
    Slayer_

    Glad to hear I'm not totally wrong :)

    +
    0 Votes
    OH Smeg

    Even the Chip Set or Integrated Circuits used in these devices are called Modem Chips or things like Modem on/in a Chip.

    Col

    +
    0 Votes
    a.portman

    Verify the Internet is down "everywhere". Then start at the connection and work in. Otherwise, start at the desktop and work out.

    +
    0 Votes
    mjd420nova

    Using just a single laptop, connect to the modem to check access from there. If that works , it's the network/server if not access there, call your ISP.

    +
    1 Votes
    jtuck004

    the boss's nephew, who tries to download the entire Internet onto his Chromebook and complains that "the Internet" is down 4x a week. And while you are in your chair, take a glance out the window and make sure there isn't a backhoe working next to the service entrace for your T1.

    With that out of the way, take your laptop and, listening for the sounds of grumbling from other users as you make your way to the main router, (so as to make sure it is really down for everyone). Then, by looking to make sure a light is on at the module where your T1 plugs in to the router, and thus making sure they haven't tripped the breaker, again, by plugging the coffee pot into the same receptacle, again, see if you have service to the switches by plugging in your laptop (into which you have hardoded an IP address and DNS server) to see if you get access at the router. You do have a straight-thru cable with you, right?

    Oh yeah, and if you find that Internet access is being provided by a modem, you aren't in as large an organization as you think.

    +
    0 Votes
    mamccfts

    I would... Keep Calm and Chive On

    +
    0 Votes
    aj42617

    *Check if the issue is 1 device or network wide?
    If network-wide this is an issue then reboot your router and/or Firewall, wait 5 minutes and check for connectivity.
    *Then contact your ISP for confirmation of connectivity.

    +
    0 Votes
    Ryalsbane

    ...would continue to function. I would remain calm and focused due to having planned for this contingency when I designed (or redesigned when I took over for the previously ousted Network Manager/Admin) and implemented the network infrastructure to include redundant failover paths to the internet. My customers (company's employees and management) would not be aware of a failure until I announced it. I would know of the failure from the 24/7/365 monitoring and notification subsystems in place for this purpose.

    Assuming a basic go/no go monitoring with notification subsystem and that the local network is still functioning, I would do as lucvdv@ suggested. To get a quick, assessment of the scope of the failure and where the potential failure point is I would perform a traceroute to multiple internet destinations. The results will quickly tell me if the problem is with my local premises equipment, between the LAN and the ISP or with the ISP premises equipment. Once the scope of the problem is determined, if the problem is with local premises equipment I follow a reiterative "divide and conquer" process using traceroute, ping, netstat, nslookup, etc., to test for DNS and routing issues ultimately continuing through until the problem is fully isolated.

    +
    0 Votes
    OH Smeg

    But as you are not using a Modem you obviously don't have a connection to the WWW. So it doesn't really matter to you does it? :^0

    Col

    +
    0 Votes
    Alec Lerg

    We have two ISPs and a server for DHCP, if the main network or backup network is down, the first thing that we do is make the changes in the server for the user to start using the other network, until we find the solution for the problem. The main network is where most of the user are connected to, the second network we have it as a backup in case the main network goes down, but still we have a couple of user in the second ISP, that way we know it the backup network is down.
    We have KS-HostMonitor, monitoring the network so, that way we know if there is a problem in the LAN or with the ISPs. Just yesterday we had a problem with the backup network. One of the users told me that the network was really slow, so I connected my computer to that network to realized that in fact it was really slow. In speedtest, it gave me less than 1MB and I ping to 4.2.2.2 and it gave me around 700ms in response. I contacted the ISP and they asked me to connect directly to the modem, I did that and it still was slow, after that they realized that the problem was on their end, and finally they replaced their equipment.
    Having a backup network is what has save our company and being able to do the necessary changes fast by having the DHCP server.