Weird RRAS Issue

I have a Windows 2003 server that is running RRAS and acting as a VPN server. Everything works fine and I can connect remotely with no issues. However, when I disconnect my VPN session, if I wait about 10 – 15 minutes and try to connect again I get the error 678 stating that the remote server is not responding. If I go to the server there are no active or hung sessions and no errors logged. If I restart the RRAS service I can connect again but once I disconnect after about 10 – 15 mins the cycle starts all over again. Has anyone else had this issue?