Hill cipher is developed using simple linear transformation. It is vulnerable to known-plaintext attack and there exist several methods in the direction of linear transformation to overcome the problems. HCM-EE is an improved method using Eigen value but it is linear and too many mathematical operations. In this paper, an attempt has been made to introduce nonlinearity to the linear transformation based cryptosystem using byte substitution over GF and a variable length sub key groups. The performance evaluation of the method is also studied and presented.