Correct. Many of us don't care too much what persons do as long as it is legal. Moral is an intellectual, indestructible ribbon: You stretch it as far as you believe you need it. Any observer does the same; judgement is injust by default because too often we do not care about the needs and beliefs of others.
On the other hand: You can't do EVERYTHING here, too. Surely not. If corruption is involved, everything gets sticky and tricky. If, as mentioned above, you make it too apparent that you have something others don't but want, normally it is going to get difficult for you. Envy and a rotten kind of self esteem will keep a balance.
So at least _most of the time_ people try to keep distance to other ones' private lifes...

Greetz from Germany