Exchange 2003 Queue

Here is my problem. The Queue, for my domain, on my exchange 2003 server is building with bogus/spam emails. Since the users they are addresses to don't exist my queue starts filling up - eventually it gets large enough and email stops coming in and the sender will get a "timeout looking for something" or undelivereable message.
I have just installed this server a month ago, dual 3 GHz processor 2 GB memory 2003 server OS running as a member server in a 2003 domain. I had an old 5.5 server. Everything seemed to work fine for a few weeks until the queue size grew too big. I did take the * out of the address space and only listed the domains I am responsible for - that seem to help keep the bogus queues out of there - I had 1500 at one time. But my legitimate queues are the problem now. FYI - Last night I installed Microsofts Intelligent Message Filter and this seems to be catching a great deal of spam but my queue is still growing with spam emails for users that don't exist. Any help would be greatly appreciated.