I must admit my experience is different. I'm getting endorsed by people that really knows me and only for skills they know me about. I'm doing the same: only endorsing people for their skills I can account for.
I got the same felling than you, when I receive my first endorment. But decided that, to be effective, it should be done in a trully way.