Remote Desktop Access for users to a File server

I am running Server 2003 and Active Directory 5.2. My issue is that I cannot get users to be able to log in with an RDP session without first making them a member of the Administrator group. This is disastrous from all angles. I need to find out how to get them in without them in the Admin group? I have tried editng the RDP properties and also the RDP user group and none of these options worked. I look forward to hearing some replies.