rdp sessions disconnecting

I have servers that we connect to via rdp sessions. they are on a site to site vpn. some of the sessions connect and stay connected but others disconnect and then it takes a while to reconnect. I can connect to these from rdp sessions already connected thru the vpn. can anyone offer ideas on where to start troubleshooting this issue?

Thanks
Frank Colvin