To protect software from unauthorized use, registration key is frequently used in the field of permission control. For the security of such a system, the key issue is to design a robust algorithm. The authors propose an enhanced scheme of hill cipher based on variable modulus and algebraic alphabet to protect software copy, which uses tridiagonal matrix. Since the modulus is not a fixed number and the algebraic alphabet depends on a variety of choices, the cipher is hard for an adversary to break. All of the choices of the modulus, the algebraic alphabet and the key matrix depend on the machine fingerprint of the buyer.