Not slow but slooooow
I agree with your experiences. I used imap for both gmail and company account hosted on gmail with Outlook 2003 and it was OK but a bit slow, however there were many stability issues of Office 2003 on Windows 7 which forced me an upgrade to Office 2010.
Outlook 2010 functionality is much better overall than 2003 but IMAP is much slower. Yes it can be configured to work offline, but then that gives problems when moving emails to different folders, etc.
My solution was to use POP and work locally. Then when travelling i use my phone to connect to gmail imap to check latest emails before i can fully connect and download on to my laptop with POP...of course i keep all emails on server, and Gmail is very good in this respect to allow both imap and pop to co-exist on the same account.
I also use connector for 1 live.com account, and it is much quicker, but we do not want to migrate all our accounts to be hosted by MS office / live just now, it is too much work versus benefit and we have other priorities at the moment.