Networking

Secret agents: Make SNMP work for you

Blogger Mark Underwood lays out the ways you can use SNMP agents to monitor network devices, and even set it up to send software alerts as well.

Out there, working for you, are agents. Feed them a little port UDP/161,162 and they'll deliver a dossier on many network devices, in the form of a Management Information Base (MIB).

Just got hired after the last network administrator got promoted to CIO? Grab a free network management tool that has an SNMP (Simple Network Management Protocol) agent listener (SpiceWorks, Net-SNMP, NetXMS, Nagios, Zenoss and many more), then head over to the local Wi-Fi-enabled coffee establishment. Chances are good you'll have charts and diagrams to visualize what you've gotten yourself into.

SNMP considerations

Here's a list of pros and cons for using SNMP agents, which I'll discuss in more detail below.

Pros
  • Intrusion tripwires
  • Quick network overview
  • Indispensible for switches representing  single point of failure
  • Proactive warnings for failing hardware
  • Enable performance monitoring
  • Detect software failures and anomalies
  • Best practice for industry standard, interoperable device descriptions with ontologies
Cons
  • False Positives
  • Log management
  • Too much information; can't see the forest
  • Complex monitoring environment
  • Configuration Management
  • Agent authentication and default public settings
  • Multiple agent message formats

SNMP basics

The basic notion of SNMP is that of an agent-based notification system. Each device, even many low level switches and printers, is equipped with an agent ready to do your bidding. The notification, or "trap," can be generated by an agent developed by the device manufacturer, or listener software can monitor systems for specific events, such as particular items of interest in an event log, and send traps to an SNMP trap handler or other network management tool.

SNMP can be thought of as one framework within a number of overlapping frameworks that include Microsoft Windows Management Instrumentation (WMI), the Web Based Enterprise Management (WEBM) and the Common Information Model (CIM). CIM has evolved into an entire object model that DMTF describes using graphical language taken from the Unified Modeling Language (UML).

SNMP does Windows or Linux

Microsoft has fully embraced the CIM model in WMI. For example, open a command window on many Vista, Windows 7, or Server 2008 machines and type:

winrm enumerate wmicimv2/Win32_ComputerSystem
The tool will list a machine's basic hardware information such as the motherboard manufacturer, but also Domain membership, status of the administrative password, server roles, current user name, machine name, boot options, and more. Using WMI, you can deck out the walls of your cubicle with ample justification for upgrading the server farm. Figure A shows such SNMP-enabled charts. Similar monitoring capabilities are available for Linux. For instance, the free WebNMS product implements an SNMP agent but also offers management through HTTP.

Figure A

Click to enlarge.

Windows Graphs from SNMPBOY.MSFT.NET

Worth the effort and cost?

The hurried and harried network administrator would be right to question how much effort to put into studying SNMP and related topics. The Distributed Management Task Force (DMTF), primary custodian of knowledge about SNMP and related topics, insists that its tutorial documents are suitable for "management application developers, instrumentation developers, information technology managers and system administrators." That may be over-reaching. Scott Neumann of the CIM Road Map Task Force describes CIM as "the most developed and widely accepted model for describing an electrical network." That said, the subject is as deep and as complex as big or heterogeneous networks can get.

SNMP for software?

Here's where the fun begins. SNMP agents are not only for physical gadgetry. For instance, Oracle Enterprise Manager (OEM) can be tweaked to respond to alerts from Oracle VM, Oracle Database, or Fusion Middleware. The use case Oracle offers is its Contact Center Anywhere (CCA) application. Oracle walks prospective SNMP users through useful telephony-related traps, but also straightforward problems such as software license failures, "Malicious Call Trace," Automated Call Distribution Voice Mail, etc. These examples show how an application could be engineered to help managers understand its performance, to automatically escalate certain conditions, or to implement enterprise-specific workflow. These could result in exciting improvements in the way software is designed.

Be advised that there is risk in this Spy-vs-Spy world. SNMP's original designers were a trusting lot, and security seemed to have taken a back seat to disclosure.  SNMP "community strings" function as passwords between the manager and the agent. The community string appears in every packet sent between them. Don't risk having your SNMP agents become double agents. Don't accept the default values of "public" or "private" for community strings. "Private" is especially problematic as it may permit an attacker to modify a device's configuration. When it makes sense to do so, and when the device allows it, limit which IP's are permitted to access SNMP agents. While not all network devices support it, SNMP Version 3 features improved agent encryption, which reduces the risk of man-in the-middle network attacks that could not only discover how to get out of your DMZ but could potentially reconfigure devices.

Buyer beware tips

A comprehensive buying guide is beyond the scope of this brief post, but here are a few important tips to get you started:

  • Keep in mind that even a small network will have hundreds, even thousands of "devices." If pricing is based on a device count, round up. Way up.
  • "Automatic Network discovery" is great in principle, but it assumes everything is going your way - i.e., that both agent and manager can see one another.
  • There are hosted as well as internally managed solutions for SNMP monitoring.
  • Your time investment will add up. It won't necessarily be all in one sitting. Prepare to invest serious time in making use of SNMP alerts. If you're on a project schedule, the community / commercial options can be a way to get help without a big investment.
Recommended readings

  1. Whemsolutions.com tutorial on the DTMF Common Information Model.
  2. SNMP Penetration Testing Technical Note from SANS Institute.
  3. TCP/IP Guide's entry on SNMP Version 3 (SNMPv3).
  4. Cisco's Guide to SNMP.

Get the PDF version of this post here.

About

Mark Underwood ("knowlengr") works for a small, agile R&D firm. He thinly spreads interests (network manageability, AI, BI, psychoacoustics, poetry, cognition, software quality, literary fiction, transparency) and activations (www.knowlengr.com) from...

27 comments
david.g.white
david.g.white

SNMP is a great tool for monitoring the status of network devices and devices connected to a Network. . No Network 'expert' would consider a sizable installation without a management tool that supports SNMP. . No SNMP installation would be 'proper' without ACL's to control what systems can utilise the SNMP facilities. (i.e. each 'agent' should be configured to only allow remote access from pre-defined management system addresses). . SNMP V1 or V2 or V3 - all depends on what the devices will support. . An access list on a Network device to only allow SNMP access from a specific IP address (or range) improves the security. . The issues with SNMP are different between Network devices and Network connected devices: . Network devices can be re-configured using SNMP sets with a RW community string (ACLs provide protection against this) . Network connected devices could also be re-configured, but running a MIB walker against a host system can give up all sorts of info that you would want to keep hidden (some UNIX implementations for instance will provide a full list of usernames on the system) - Windows system will also provide details that a hacker / cracker / freaker / etc. will find invaluable (generally ACLs can be deployed to protect this). .

Neon Samurai
Neon Samurai

Has anyone thoughts on the security implications of snmp? I've looked at it a few times but always get hung up on the fact that it's a clear-text protocol. Any sniffer on the network delivers a complete detailed data flow to an attacker thanks udp 161/162.

Neon Samurai
Neon Samurai

Thats sort of where my paranoia hits.. specifying an IP and MAC address is a natural ability of most OS and snmp read-only and write names can be plugged out of the network traffic with the older protocol versions. My knowledge is limited in this area though as home devices are horridly out of date and I keep my fingers out of the Cisco boxes at work.

knowlengr
knowlengr

These reminders and others posted here are good. I do find it odd, though, that defending against end user outages, including application outages -- a far more common job risk than security concerns -- isn't front and center. That said, it sounds like a post on SNMP security -- here or elsewhere on TR -- would be of interest to folks?

tmcclure
tmcclure

Like anyhting else you have to wiegh the risk. SNMP is very useful to me. And good security policy and practice is key to implementing it. A couple of simple rules go a long way to help secure it. Don't let SNMP traffic outside your network and don't let folks plug stuff in to your network without knowing about it.

seanferd
seanferd

but that is exactly why it gets disabled on every home system I am asked to service.

jhoward
jhoward

I will say this again - SNMP isn't the security threat here. You can argue that SNMP is plain text, easy to spoof etc. all day long but it still doesn't change the fact that for an intruder to get this information you have been compromised already. In the end if a malicious attacker has access to your SNMP packets they have access to other much more sensitive areas of your network. In other words I would be more worried about how someone could possibly gain access to capture or spoof SNMP traffic on my LAN or private network than the actual SNMP traffic itself. That being said I still stand by locking down SNMP as much as possible by using SNMP v3, ACLs (SNMP config and firewall), disable SNMP write access and limiting the OIDs a community has access to. This type of configuration practice should be standard for any network service though.

Neon Samurai
Neon Samurai

I'd be interested to read details on the potential security aspects outside of physically separating management and user data networks.

Neon Samurai
Neon Samurai

For regular use, that's just a great idea all around. What I'm thinking of is the attacker that drops a modded Iphone or other tapping device into the network. Consider something in-line like a transparent proxy and you won't even be seeing it's additional MAC in your IDS. Luckily, our network is still small enough that guests are obvious and something like Munin or Splunk with a client side reporting program and encrypted network traffic remains a viable alternative.

knowlengr
knowlengr

Home users may be more tolerant of acting as purveyors of failure events than one's supervisors in enterprise environments.

Neon Samurai
Neon Samurai

Though, "Hacker" suggests someone who has permission to be mucking about inside the network which is why I specified "attacker" though "cracker" would work and the truest term would simply be "criminal". But semantics aside, I was kind of hoping to be corrected by someone telling me there was a way to lock it down finally. Seems snmp will be kept minimal on my networks. I'll have to look into the few places where it seems to be a required component.

jhoward
jhoward

The gratuitous ARP only goes out when the NIC interface swaps on bonded interfaces. This is why you have to setup a virtual MAC. The ARP goes out saying the MAC/IP is now on a different interface so every other device updates their ARP table and traffic for that IP/MAC and... It is simple in how it works however anyone with physical access to the network potentially can steal the IP of any other physical device on that network. In some cases you do not even need physical access for this but simply access to a compromised machine and the ability to take the target machine out of service. As I said before - if you are at risk of someone seeing or spoofing your SNMP packets you have bigger issues.

Neon Samurai
Neon Samurai

I was actually thinking of the general "read" function which I'm guessing is the SNMP Get. I generically refer to most network traffic as being sprayed all over the network due to how TCP/IP packets move through whatever series of nodes gets them to the destination. The only time I wouldn't consider network packets sprayed all over the network is with some sort of encryption since packet data is then limited to the sender and recipient regardless of what nodes it crosses in-between. (well, in theory anyhow, there is some weak encryption out there) When your management box makes a get request, that response packet still has to travel over the network. Get a tap in-between and you get the same info the management box is asking for. If I can suck up read and read/write SNMP community names as a third party then by extension, I can get the contents of that chatter. Granted, doing this in a passive manner is harder then the simple but noisy ARP methods. osX was not related to SNMP, just an example of something spraying packets all over the network. That's actually a UDP thing which seems to be related to Smaltalk. The short term solution is to accept the network noise but when time permits, I may look into the machine config further and see if that can be stopped. SNMP is something I need to read more about in general though. I've heard tell of traps but haven't worked with SNMP enough to claim much authority on the subject.

jhoward
jhoward

I think we are talking about 2 different aspects of SNMP. We only query for status using SNMP get and have SNMP traps disabled because of the network spray you mention. I will agree SNMP traps clutter your network however the frequency and number of SNMP gets is completely controlled by the administrator. The amount of time it takes to reign in SNMP traps is not worth it. We generally poll SNMP stats on equipment between 2 and 10 minute intervals - switches we generally poll more often as these are usually where we catch issues first.

Neon Samurai
Neon Samurai

I think it could offer a more passive way to do it at least. Flipping a switch over to dumb-hub mode isn't quiet nor is arp poisoning. There are many other ways to get the same information but SNMP can provide a more passive way depending on where the tap is inserted. For me it's more a matter of ongoing pruning. osX sprays packets all over the network so I'd like to stop that behavior where it's not required. Printers seem to rely on SNMP so I'd like to confirm if it's required or remove it. SNMP would benefit some of my more remote systems but there's no way I'm spraying that over public networks when the limited number of machines is easily supported by Munin for status and alarm triggers. I also admit to a personal bias against any cleartext protocols when things like HTTP and the other granddaddy prots are way overdue for replacement. (My bonded NICs are not spraying arp out over the network. I may need to do some reading on that.)

jhoward
jhoward

Please don't take my above post the wrong way - I am not advocating unprotected traffic over a LAN or private network - I was merely pointing out that the security risks being associated with SNMP are not specific to SNMP but to network security in general. Now - is the information obtained in SNMP packets useful for a potential intruder? Absolutely. But not much more so than gratuitous ARPs and ICMP pings. For example: Linux servers utilizing bonded NICs (Cisco hardware uses similar methods) use gratuitous ARP messages to facilitate automated NIC failover. ARP messages are super easy to spoof and provide arguably more information on your network topology than SNMP. How is the threat level of this type of attack different than that of SNMP? I guess the point is as others have said - the pros outweigh the cons by a lot when it comes to SNMP for my purposes. Others will have to make their own judgment call.

Neon Samurai
Neon Samurai

My general thinking is that security needs to be comprehensive beyond a crunchy candy coating. Far too many places think; "well, we have a firewall on the outside so we can leave everything wide open on the LAN." So, what happens if your perimeter is breached? Do your individual nodes protect themselves? Does your internal network traffic protect itself? How does an attacker get inside the crunchy shell and what can they do then? How can they manipulate traffic to negate switches and such? If they can't get through the network uplink or into the office phisically, can they hit one of my user's devices outside the office and what access does that give them? With SNMP, the write permissions may be locked down. Read permissions may only respond to given ACL members. Those packets are still flowing over a potentially manipulated path though. This means a complete view of the network; inventory, health/weaknesses, early warning against discovery, monitoring and probing the IT reactions to problems - all the same things that make SNMP beneficial to the IT staff plus monitoring of the IT staff. Now, an extra notebook with two NIC stuffed inline in your server room may be noticeable but how many other ways are there to tap the network? An Iphone or similar device modded and stuffed behind a printer is pretty subtle; in a ceiling tile even more so. Consider the physical security and custom hardware from Chris Nickerson OWASP talk "Red Team Testing". http://vimeo.com/7593801

seanferd
seanferd

may find that they are not so tolerant, unless the protocol is further improved. Otherwise, I agree, seeing as an enterprise environment actually has some use for it. It is just one of those things that should not be on by default, since it is unlikely to be properly configured and accounted for in environments which make no use of it.

Neon Samurai
Neon Samurai

We don't have a fast hardware turnover but I've noticed two of our newer hardware bits showing snmp v3 settings. I'll have to read more about it and look at what they can do. A dedicated network over security pair type medium instead of shared wires seems like a big benefit to the setup too though. I remember running isdn over security pair log ago. I guess it's like CIFS/SMB in that it's become essential for the network though the protocol has room to mature.

jhoward
jhoward

We manage hundreds of routers and switches for our customers over managed private connections (T1, DS3 etc.). If we didn't use SNMP we would not be able to do our job so for us it is extremely important to know the risks/rewards and why I gave the brief list above. Just like setting up an SMTP server - by default it is very unsecured and open to many intrusive and malicious actions however it is up to the system administrators to lock it down. Looking back at NeonSamurai's post I wanted to mention the firewall ACLs are important but probably more so are the SNMP access control lists themselves on the device you are monitoring. Limiting the networks and OIDs each community or group has access to and from is also a highly recommended practice although somewhat time consuming depending on the device.

knowlengr
knowlengr

The offsetting benefits of being proactive prevail, especially for harried, multi-dutied network managers.

Neon Samurai
Neon Samurai

I can see it's benefit in larger organizations, I just wish SNMPv4 turned up with a properly encrypted protocol. Telnet is a great example of a cleartext protocol that needs to be taken behind the barn and shot in the face. These days, there really is no excuse for a vendor to deliver a device without ssh support; it's not like OpenSSH costs them more than a telnet server to implement. Even 100$ home user devices have no excuse; I really don't like the extra steps I have to take with my NAS box when solving it at the source would be easy for Linksys. Disabling write access and blocking SNMP packets at outer firewalls are ideas I'd also considered so it's good to know I'm on the right track still. One could also implement a secondary admin-only network as is often done with ILO connections on rack hardware. It at least provides a physical separation from the regular traffic wires.

jhoward
jhoward

To be realistic many of the security concerns are related to incorrect configurations and security issues outside the scope of SNMP. The risk of a person or program with malicious intent getting valuable information from SNMP is arguably the same as using telnet across the LAN to access your routers and switches which unfortunately is still a common practice (not meant as a flame war just an example). Personally the benefits of SNMP outweigh the risks for us but your mileage may vary. There is simply no way we could offer the level of service we do without SNMP simply because it allows us to be proactive with our network devices. The main concern for us is preventing man in the middle attacks which aside from SNMP mean making sure our internal network is just as secure or more secure than the external. Tips for using SNMP to limit exposure: - Don't use traps. They are too easy to spoof - Disable write access. - Use SNMP v3 and access lists to limit exposure - Limit the OIDs a community has access to to limit the extent of information someone can gain - If you need SNMP across a WAN interface only allow it through a secure tunnel

Neon Samurai
Neon Samurai

I don't see faking the snmp responses as too hard a trick for someone who's already cracked into the network through outer layers of protection. Encryption does not just add privacy but also authenticity which is lacking in pre-v3 versions if not the v3 version also.

knowlengr
knowlengr

Indeed, v3 addresses security issues, though plenty of devices don't talk v3 at this point. There's argument over this, but there's a case to be made that SNMP alerting is also a hacker's worst nightmare. A coordinated set of SNMP alerts makes for a great honeypot.