Constant FTP Disconnects

What is it that causes me to loose connection with any FTP site after awhile? Whether I send dozens of little files or large ones, I get timeouts. I watch my files up/download fine, then they slow down and stop and I get timeout errors. I have tried 2 FTP clients and connections into 4 different FTP sites of people and businesses I know. Sometimes it timesout but I still stay connected in the FTP site.
This has been happening for months and I just moved into a brand new construction home, new jacks, and my connection (according to TimeWarner) is spotless and clean at (supposedly) 5M down and 300+k up.
Even with Windows explorer. If I VPN into a site and start copying/pasting, after awhile, it timesout. My VPN conenction is still active but the transfer failed.