The Microsoft Experimentation Platform wants to take decision making from the Hippo (highest paid person’s opinion) and take the guesswork out of project planning. The goal of General Manager Ronny Kohavi is to build a system that would map user activities using machine-learning algorithms.\r\nRather than make changes on the run, ask your customers first.\r\n
Read Mary Jo Foley’s interview with Ronny Kohavi.
Measuring the success of a concept should be simple according to this diagram. Take two versions to test, a control–usually the live version, and the treatment which is usually the new idea. Then you simply collect the data and analyze. \r\n\r\n
Following is from a white paper from Microsoft Experimentation Platform: Practical Guide to controlled experiments on the Web: Listen to your customers not the Hippo.
Ads were palced at the bottom of this shopping section to determine their worth. Clickthrough rate was 0.49 percent lower and page views were down 0.35 percent on the page with extra ads. Apparently not significant but the expected revenue was less than the loss of revenue from lost clicks, so the project was scrapped. \r\n\r\n
Click on the image to enlarge.
This experiment ran in Windows Marketplace/Game Downloads. The Solitaire game had a higher clickthrough–61 percent preferred Solitaire.
High-contrast text was tested against softer colors. The high-contrast page drew 0.9 percent more queries per user and 3.1 percent more ad clicks.
Feeback A puts everything together while Feedback B is a two-step process–a question following the ratings. Feedback B got more than double the response rate.
This feedback is also a two-stage process. It outperformed Feedback B by 3.5 times.
