Remote desktop disconnect

Running windows server 2003. Trying to connect in the office from win xp machine and also tried connecting from a windows vista machine.

Get the dreaded The client could not connect from the remote computer. Remote connections might not be enabled. Or the computer might be to busy to accept new connections. It also might be a network problem.

Weird thing is. I can connect from home just fine. It is only when I use the computers in the office. I get the disconnect message.

Any suggestions?