Normally, when you install the High Encryption Service Pack to get the 128-bit encryption on your Windows NT 4 server, you leave it there. That is what I thought until my company started implementing Oracle’s Ipayment server, which handles credit card processing for Oracle-based accounting systems, and I found that there was a compatibility issue with the 128-bit service pack. So what do you do to get your server back to 56-bit encryption? In this Daily Feature, I’ll give you a few suggestions.

Check your backups
Even if you back up your server on a regular basis, chances are you don’t have copies of your backup tapes from when you initially upgraded your Windows NT server from 56-bit encryption. If you did keep a copy, you’re in luck. Restore your backups and you’re in business. If you don’t still have a 56-bit backup of your server or aren’t sure if you do, don’t panic. All is not lost yet.

Check your primary partition’s file system
If your primary partition is FAT, then you can boot from a DOS disk and manually copy the files that need to be downgraded. Boot your server using a standard DOS disk that has the necessary DOS drivers for your server’s CD-ROM. When the server boots, place your original Windows NT CD-ROM in the server and copy the following files to your server’s C:\Winnt\System32 directory:

  • Ndiswan.sys
  • Ntlmssps.dll
  • Schannel.dll
  • Security.dll

(If you used NTFS, continue reading to see what the best choice is for you to address this problem.)

Install the Standard Encryption SP
The next step that you can try is to apply the Standard Encryption version of the Service Pack installed on the server. If your server runs Service Pack 5 or earlier, you can install the Standard Encryption Service Pack over a High Encryption installation. If you’re running Service Pack 6 or later, you can’t. Microsoft changed the Service Pack’s setup program to prevent this.

If you’re successful in applying the Standard Encryption Service Pack when your server reboots, check the four files listed above to see if they show as the export version (40- or 56-bit) or U.S./ Canada only (128-bit). Look for each of these files in Windows Explorer, right-click on the file, click on Properties, and then click on the Version tab. If you see Export version, the file is the standard encryption version.

If you have one or more files that insist on remaining at the U.S./Canada version—Schannel.dll will most likely be the problem child of the group—the next step is to extract the Standard Encryption Service Pack to the server. One reason that Schannel.dll may be a problem is that it is the primary file involved in the encryption process and is also used by applications other than NT itself, such as Internet Explorer.

Go into the I386/update directory and edit the Update.inf file. Look for the [CheckSecurity.System32.files] section and put a semicolon in front of the line that references Schannel.dll. Doing so disables the version checking that normally occurs as a part of the Service Pack installation. Close the editor and reapply the Service Pack. If the file still won’t downgrade to the standard encryption version, you have a couple more options to pursue.

Perform a dual install of NT on the server
If everything else has failed, you can attempt to install a second copy of NT on your server and boot from that copy. By doing this rather than completely reinstalling NT from scratch, none of the files you tried to downgrade in the original copy of NT will be active or affected. I strongly suggest that you have at least one good backup of the server, if not two, before proceeding further. Also make sure you print out the Boot.ini file on the server.

Of course, this option assumes that you have sufficient disk space to install another copy of NT onto the same drive and that you will apply the Standard Encryption NT Service Pack. Once you have completed this parallel installation, you can copy the four files previously mentioned from the newly installed copy of NT to the one that you will boot from.

Once you have copied the necessary files to the original copy of NT, edit the Boot.ini file, removing the lines that were added to what you documented before starting the install process. Reapply the hidden and system attributes to Boot.ini and reboot the server. You should now come up with the correct version of the files.

Force a manual downgrade of the files
The easiest way I have found to downgrade files that won’t seem to cooperate through other means is to get a copy of NTFSPRO from Winternals. You first create a DOS boot disk and then install NTFSPRO on the server. Part of the NTFSPRO installation process will create two disks. Boot the server from the DOS boot disk and then run the Ntfspro.exe from the first disk that was created from the NTFSPRO installation on the server. You should see all the NTFS drives appear by drive letter.

You can now replace the files that wouldn’t downgrade through other means and reboot the server. The whole process, from start to finish, should take less than five minutes with this option.

As you can see, there are several ways to get the files downgraded to the version you need. The solutions offered above can apply to other situations and not just the encryption scenario you have just read about. If you elect to go with the NTFSPRO product, take a look at the company’s other offerings; you may see some other tools that need to be added to your bag of tricks. If all else fails, and you still must go back to a 56-bit version of Windows NT, then you must reinstall Windows NT.
The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.