When I first started using the beta of IE 7, I had the phishing filter turned on and it crawled. I think several page loads got rear-ended by a glacier.
I turned off the phishing filter, which MS recommends against, and the speed issues went away.
I haven't tried the phishing filter since, despite repeated warnings from Vista and IE 7 that I should turn it on for all user accounts.
So far as I can tell, the phishing filter isn't really ready for use. This may have changed since I tested it several months ago, but from what I hear, I doubt it.
So, if IE 7 is really slow for you, try turning off the phishing filter and see if that helps.
I think the phishing filter databases may be a product that could be installed on a local machine and have daily updates, like antivirus, but I think performance over the web may be too slow. It basically has to be a database search, and since it's comparing text strings, it's going to be relatively low performance. Add in millions of concurrent users, and you have a serious database use nightmare situation. Make it a simple local database (like AV) and it might peform well enough to be a real product.
MS would probably be better off adding it to their OneCare product than making it part of a Web-based solution in IE. (Should be easy enough to create a Firefox plug-in that would work that way too, I would think.)
I could be wrong about that. Maybe it can work over the Net. But I don't see that being the reality at this time.