Wireless and mobile communications are having great features and is attractive among users as well as service providers. Unlike wired, wireless networks provide anytime, anywhere access to the users. Masquerading and eavesdropping are major threats in mobile Communications. To provide protection in communication, the authors require enciphering of voice message and authentication for subscriber with the communication network. This paper presents a complete study report on authentication protocols and its application on mobile communication systems like GSM, GPRS and UMTS. In addition, they describe the encryption and authentication algorithms that are used in these architectures.