would have been "excuse me, but I didn't realize I was talking to you", and then walk away.
It is perfectly acceptable to be rude in return of someone being rude to you, and eavesdropping on your call.
Or you could have taken the more direct approach and told him to go bugger off. Works either way.....
Keep Up with TechRepublic