In my position I have to make a few site visits (different offices and different buildings) most days, and the side tracked conversations do waste a lot of my time. Because I don't see some of these people frequently they tend to want to play conversation catch-up. I sometimes wonder if they have as many conversations throughout the day with other people. It's not the 10 or 15 minutes wasted with one person, it is the 8 or 9 different people that sidetrack me along my way. But I can only attest to my experience in this arena.
As for the network side of the distractions, this I can speak too from a more knowledgeable point of reference. The biggest distraction I see is from email. Not just the email itself, but its content. You can see when an interesting email has circulated from the impact on the network. First there is a spike in third party Webmail activity (as well as our internal email), which is usually followed by a spike in hits on a specific site (YouTube is a big one). Sometimes these circulations have been so bad in some of our remote sites that our network has taken a significant hit. Although it circulates fairly quickly and subsides, it still impacts everyone on the network for 30 or 40 minutes (whether they are included in the circulation or not). So I guess in a way this is a form of social networking, as it is rarely a work related email that spawns this activity.
We have since had to block non-work related webmail sites (whether the users know it or not, this probably annoys me more than it does them).