Acces a win2003 server with RDC through 2 diffrent ISP

HI,

I want to be able to access a win2003 server with remote desktop through 2 different ISP’s. I have two routers connected to a switch. The server is connected to the same switch. On the LAN side the routers have the IP addresses in the same class with the server. I have configured 2 default gateways on the server and 2 dns address to match the ISP’s. Both routers have configured nat on port 3389. I can connect from one ISP but from the other one it doesn’t work. I tested the connections with a win xp workstation, instead of the server and it worked fine. Can someone help me please!!

Thank you!