Workgroup access to Domain

I have Windows 2003 server (AD)installed and runing and i have noticed that if someone knew the user name and password of of a domain user will be able to access my network having all the assigned rights to this user. Does anyone know how to fix this issue.