Why both 32-bit & 64-bit programs installed?

I noticed today there are two folders of Program Files in all the new Windows 7 64-bit laptops. One for 32-bit and one for 64-bit they said. I wonder why. I even see two versions of Internet Explorer on the Start Menu.

To what I have understood 32-bit programs run on 64-bit machine. Why do we need both installed? If 64-bit counterparts are available they can simply have 64-bit programs only installed. Quite a few of them repeated in both folders though the contents might not be identical.