Connectivity issue with my server on a particular port

I have a windows 2008 server which client connect to over the internet, but after sometime some users will be able to connect while others will not connect successfully.This will stop once i restart the service and every body will be able to connect again. but i want to stop restarting services because the down time gives my business a bad name