Quality of voice delivered over packet networks is affected by various factors such as packet loss, end-to-end delay, packet delay variation (jitter) and codec bit rate. Different approaches and models predict speech quality as a function of such impairments. In order to ensure a continuous play-out of voice transmitted over a packet switched network, jitter buffers are commonly used to counter jitter introduced by queuing. In this paper, the authors propose a new adaptive jitter buffer algorithm based on optimizing the predicted voice quality. The algorithm consists of an adaptive play-out mechanism based on the extended E-Model taking into account packet loss pattern and a time-scaling technique relying on a speech classification mechanism embedded in the decoder.