network speed

I guess that the server is working like proxy and has two network cards. Use a program to measure throughput out (MRTG or other) from your ISP. After you have a clue about what you use, the next step is what program use this.You can use netstat -nao from command line to figure what program is on that computer.
Could be an update for antivirus, etc.

did you install some network software base like anti-virus. etc?, could be that during those time, a scheduled update is running casuing to slow down the traffic. You can check the server performance during that time, type "perfmon" from run command, and run the proper counter to diagnose the problem.

1) at 4PM all your workers are checking the weather, road conditions or surfing for porn before they head home??

2) At 4PM the high-school hacker who has compromised your server gets home from school, and all his buddies are downloading the warez/gamez/porn he uploaded the night before?

3) But seriously, look at the server event viewer system and application logs to see if anything special is going on at that time. When does backup run? Look at Windows task manager at 3:50 PM and again at 4:10 to see what tasks are running, and what CPU utilization you are at.