Trouble with Terminal Services

Can anyone explain why one minute we can Terminal Service into our servers anywhere and the next minute we can only Terminal Service in when our VPN is up & running?