Well now you're asking me and I may give you yet another set of conflicting information.
Yes there is a certain vulnerability there, as you have port 3389, a fairly common and well known port, exposed to the internet. One would-be perp would have to find that one exposed port on your IP, and then try to exploit a weak password or some yet unknown term server vulnerability.
Weak passwords are probably your biggest vulnerability in this setup, so if you're going to keep your configuration that way, then make sure that you give TS remote logon rights to only those usernames that require it, lock down the term server so that even users with TS logon access have only enough access to do what they need to do on the term server, force good passwords for at least those TS user accounts and any administrators, and force password changes regularly and/or any time there is significant staff turnover.
To answer the part of the well known port, you can change the port that the TS uses to something non-standard. Change the port number in your virtual server setting accordingly, and a simple change to how the users connect to your TS and you can use a different port number than 3389.
You did say that you have SBS, so you could undo the exposure of your term server and have the users access the TS via SBS RWW, but I'm not convinced that's any more secure, plus there's an extra port exposed (would need 443 and 4125 virtual server'd to your SBS, instead of 3389 to your TS).
Lastly, if you wanted tighter security than exposed RDP or RWW port(s), then you could use VPN. I'm not convinced that the built-in SBS VPN would be much more secure since it would still depend on the users' AD passwords. So if that was still a concern, then you could go with a VPN appliance at your edge firewall that would require a specific VPN client and separate VPN credentials or some other kind of VPN authentication.
So, lots of options here.