Usually when mal-ware prevents anti-virus software from installing, it is a virus or worm.
I would start with the Microsoft Malicious software removal tool.
http://www.microsoft.com/download/en/details.aspx?id=16
While it does not detect or remove every virus or worm, it does remove the most active each month.
I download it and burn it to a CD then scan the suspect system with no network connection.
Another option is the Avira Rescue System.
http://www.avira.com/en/support-download-avira-antivir-rescue-system
This builds a bootable CD that runs a live Linux and then can scan your hard drives for a large number of viruses.
NOTE: If Avira removes key system files, you may need to use recovery media to be able to boot into Windows.
Once the system is clean, make sure that a current AV program is installed and that all patches are up to date for all programs and not just Windows. Even for dial-up, I always recommend a firewall in addition to AV software.
Chas