First of all. VPN protocols create 25-35% overhead in addition to traffic you are transmitting. This means that if you have, for example, full T1 dedicated just for you, your real throutput using VPN will be 1.5Mb-30%=~1Mb. And it's just for a scenario when nobody uses this link for anything else.
Second, you should watch your Internet link utilization. Most likely you will see that you have "slowdown" right in time when somebody download something big from the Internet. I would recommend to use something like "MRTG" package. It will allowe you to monitor links utilization 24x7.
Third, you should determine, what kind of traffic do you have on your Internet link and, may be, prioritize "critical traffic". In some cases (depending on hardware), you may even allocate different bandwidth for different types of traffic. In this case you may "reserve" certain amount of bandwidth for VPN and nobody else will not use it.
If you have more questions - send me e-mail.
Good luck.