URL’s with https downloading too slow

When I use Internet Explorer 6.0 (with 128 bit encryption). Majority of the web pages downloads really fast, however, only the url’s with https takes forever to download. I’ve tried Start –> Run %temp% [Enter} and deleted all the temp files; emptied all the cache files; moved cache from the server to my harddrive and I still have the problem. May have to check the server for the profile permissions. We are running apache with Samba. Can anyone help?