During these periods of slowness open up task manager on the server. Go to the processes tab and sort by CPU time. See which processes are using up most of the processor time.
I have seen virus scanning software that begins to take up too much CPU time. TrendScanmail for Exchange had this problem and they have a patch out to fix it.
Also check your file system virus scanner and make sure (A) that it is not scanning the M: drive and (B) it is not scanning the exchange database (edb & stm files.)
There is another thing you should check. Unless you have the Enterprise version of Exchange you have a 16GB limit on the database file size. Browse to the Exchange database folder (usually C:\Program Files\Exchsrvr\MDBdata.) Yoru main email database file will be named Priv1.edb. Also check the Priv1.stm. The combined total of the 2 of these is your database size. If you are close to 16GB you are likely experienceing slowness due to extreme fragmentation in the database.
Here is a technote for doing an offline defragmentation of the exchange database. This is the only way to reduce the file size of the Exchange databases.
http://support.microsoft.com/default.aspx?scid=kb;en-us;192185&Product=exch2k