windows server 2003 remote desktop

we are have this strange problem with our windows server 2003 and remote desktoping in to it from the outside, it work fine for 3,4 days and then starts taking for forever to load the dell background without even show the login options before is kicks the connection out later it just starts locking up and then finally just never connects to the server, to solve it requires a server restart. i cant find anything wrong in the logs in the server and our router/firewall is just reporting dropped connections and i cant figure out whats going on or causing the problem, the router is set to allow the rdp so that not the issue,