Variants versus combinations in multivariate Google Website Optimiser

When you have a multivariate website optimiser experiment running, there are two ways to look at the results - by combination, or by variant. We always recommend the combination view, and I was pleased to note at the GAAC summit yesterday that the team at Google agree.

Let's say that you are testing changes to two sections of your 'hats' page - three different pictures (a selection of your most popular hats, just your most popular hat on its own, or a model wearing the hat) and two variations on the text of the call to action ('show me the hats', or 'hat me up'). The GWO interface allows you to look at a list of all 9 combinations of picture and text, or a list of all 5 variants. You might see from the combinations page that the page showing 'hat me up' and the model together is better than the original by a small margin. There's nothing very compelling there though. Naturally, you are tempted to look further. The variants tab shows that 'hat me up' is actually a little worse than the original. What conclusions can you draw about the 'hat me up' text from this?

• it would be racing ahead if it weren't held back by the two 'hat-only'
  pictures.


• it is holding the model picture back. That's why the winning combination isn't beating the original by as much as you expected.


• it is probably somewhere between a little better and a little worse than the original.

The truth is that drawing any conclusions at all from the extra information you gained from the variants tab is not really worthwhile. You could persuade someone of any of the conclusions using the data you have - but none of them are anywhere near as useful or valid as the data you had to start with: the combination of the model's picture and the 'hat me up' text is probably a little better than the original.

One of the Google team explained yesterday that the 'fractional factorial' view (the list of variants) is provided for those that want to use it, but the 'full factorial' view is the one that they really recommend. Certainly, in our experience, we've never seen a case where studying the list of variants led to a decent actionable insight.

What is a Google Authorised Consultant?

Google provide Google Analytics and Google Website Optimiser for free. It makes sense for them to do that, because these two applications make websites better for users, which makes advertising through AdWords more valuable, which makes Google money.

However, it doesn't make sense for them to provide in-depth support for these applications. To do it right, they'd have to engage deeply with each client, understand their business's specific needs and challenges, and maintain a relationship with each client over time - and all this over dozens of countries and time zones.

Instead, what Google do is to certify and authorise private companies all over the world to provide support for these products on their behalf. As well as enabling all the aspects of support described above, this allows there to be specialists in different industries and types of website, and support companies of all different sizes, so that users of Google Analytics and Google Website Optimiser can find the right fit for their business.