If you only enable RD through group policy, but the machines themselves are still set to "Don't allow connections to this computer ," when Group Policy refreshes (every 90 minutes or so), you'll be disconnected from the machine.
Per
http://support.microsoft.com/kb/2083411 with this GP, also deploy a registry tweak to allow RD connections on the target machines. Then you won't be randomly disconnected.