Whilst this piece is well written and discusses the effects of poor testing, there is no discussion about what makes a good test in the first place.

As a professional marketer, I am all too well aware of the fact that people will

"test two things twenty times"

... instead of "testing twenty things twice"

This is crucial.

More importantly the differences are not usually apparent to those who are setting up the tests. That is how they test the same thing and *think* it is different. How it appears to the reader is however pretty well the same. Having said that, if Google tests forty different shades of blue, they have the mass audience to justify it. With a billion searches every few seconds, this kind of metric will make a difference. However slight!

So what does make a difference? What kind of thing are you looking for as a difference?

As I said, it is not an easy thing to grasp. Because it deals with the emotional realm. If you don't agree, that is fine by me. If you tell me this is rubbish, that is fine by me. It matters not a whit what you think. Because your actions are determined by what you feel *before* you think.

At its simplest, what do your readers most like - and most dislike. Understand this and within a short while a large discrepancy will emerge. It will be way beyond the realms of "statistical margin of error"* - usually by a factor of 30-50%. If the margin is less, I know that my split test isn't hitting the right spot and I need to find another aspect to measure. This sort of testing usually makes statistics irrelevant, just looking at the broad figures will tell you which of ten advertisements/questions are working.

As I say, the margins will be substantial and obvious.

*See above; if your split test is that narrow, you are not performing a real test.