To fix your Google Chat contact list, locate the Chat widget on the left and click on the grey down button in the upper right corner:

This will bring up the following dialog, where you want to click on “Your circles” and select the custom option.

From which point you can then select the people you wish to chat with, and have you in their contact list.

Give Google Chat (or your chat client of choice) a couple of seconds to poll the servers, and you will see former contacts disappear.

Enjoy the regained control and privacy.