Connection between two Clients.

I try to programm a Mobile Messenger (C++, Qt) where the Server only sends the current IP address of Phone A to Phone B and vice versa and the data is send directly between these two devices with the goal to minimize the needed calculationpower of the server.

I try to understand if that is possible:
Client A has a connection to Server S and Client B has a connection to S as well. If S sends the IP and Port of A to B, can B send something to A without creating a new connection through using the IP of A and the Port which A is already listening to?

Has someone a better idea how to solved this or is this idea in general impossible i.e. data between two smartphones always has to be send through a server?