I'm not aware of a way to do what you're asking in native exchange; perhaps there's a third party solution, but I really don't think so. Such a solution would have to STOP a user from creating new emails once they reach their limit, otherwise the outbound SMTP queues could continue to back up.
You say you have a problem with SPAM, yet typically that problem is relative to INBOUND SPAM, however you're asking about limiting the amount your users are sending? I don't understand your request, unless you're saying that your users are sending the SPAM or that your server is relaying.
You didn't mention what version and service pack of exchange server you're using, but if using Exchange Server 2003 SP1 or higher, you can use Intelligent Message Filtering (IMF) to try to reduce inbound SPAM (if EX2003, preferably SP2).