How do I deny local logon to a server?

I know it is possible to deny users the right to logon to certain systems, but I can’t seem to keep them from logging into my server. (Not that they do, but precaution is the key.) I have an NT4 server, FAT formatted c:, NTFS d and subsequent drives.

I have gone into the UserMangaer and looked at my groups, and taken away all but the essential groups rights to logon locally to the system.

However, any user on the domain can logon. How do I prevent this? What critical step am I missing?
thanks