Internet Speeds slowed after removing NAT

Hello

My Internet speeds slowed after removing NAT on my router.
I am implementing a VOIP Project that requires we remove NAT and port forwarding.
We use the same router for Internet.
Can some one please tell me what I am missing?

Even with NAT enabled when I dont issue the command;

ip nat inside source list 1 interface Serial0/0 overload

Internet remains slow.

Regards

James