Variants versus combinations in multivariate Google Website Optimiser

When you have a multivariate website optimiser experiment running, there are two ways to look at the results - by combination, or by variant. We always recommend the combination view, and I was pleased to note at the GAAC summit yesterday that the team at Google agree.

Let's say that you are testing changes to two sections of your 'hats' page - three different pictures (a selection of your most popular hats, just your most popular hat on its own, or a model wearing the hat) and two variations on the text of the call to action ('show me the hats', or 'hat me up'). The GWO interface allows you to look at a list of all 9 combinations of picture and text, or a list of all 5 variants. You might see from the combinations page that the page showing 'hat me up' and the model together is better than the original by a small margin. There's nothing very compelling there though. Naturally, you are tempted to look further. The variants tab shows that 'hat me up' is actually a little worse than the original. What conclusions can you draw about the 'hat me up' text from this?

• it would be racing ahead if it weren't held back by the two 'hat-only'
  pictures.


• it is holding the model picture back. That's why the winning combination isn't beating the original by as much as you expected.


• it is probably somewhere between a little better and a little worse than the original.

The truth is that drawing any conclusions at all from the extra information you gained from the variants tab is not really worthwhile. You could persuade someone of any of the conclusions using the data you have - but none of them are anywhere near as useful or valid as the data you had to start with: the combination of the model's picture and the 'hat me up' text is probably a little better than the original.

One of the Google team explained yesterday that the 'fractional factorial' view (the list of variants) is provided for those that want to use it, but the 'full factorial' view is the one that they really recommend. Certainly, in our experience, we've never seen a case where studying the list of variants led to a decent actionable insight.