Networks

You need to assign priority to VOIP for enough bandwidth to maintain the calls. RDP will suffer some, but apparently there is not enough bandwidth to run both at once fully. QoS allows you to decide what gets priority and what suffers, if there's not enough bandwidth. If no priorities or bandwidth allocations are set, the bandwidth is usually utilized on a first-come, first-served basis - and RDP can be a hog on bandwidth.

Another thing you might try is reducing the color depth of the server or setting a color depth limitation for RDP. Also, you can remove graphics from the desktop background on the server and use a single color. When a window closes, the image behind it must be resent, but if it's all one color, RDP can send a "flood" color instead of a long picture. The less graphics you send over RDP, the more bandwidth will be left for VOIP.