GPO disallowing File Downloads…but need exceptions

I am fairly familiar with Group Policies. I have a Windows 2003 Server DC, and clients are Windows XP Pro SP2. I have always disallowed any file downloads from Internet Explorer to prevent toolbar installs, viruses, games, etc. It’s come up now that there are actually some files that need to be allowed (.PDF, .XLS, etc) and users can’t because file download and “Save As” are disabled. How do you add exceptions to the rule??