Variants versus combinations in multivariate Google Website Optimiser

When you have a multivariate website optimiser experiment running, there are two ways to look at the results - by combination, or by variant. We always recommend the combination view, and I was pleased to note at the GAAC summit yesterday that the team at Google agree.

Let's say that you are testing changes to two sections of your 'hats' page - three different pictures (a selection of your most popular hats, just your most popular hat on its own, or a model wearing the hat) and two variations on the text of the call to action ('show me the hats', or 'hat me up'). The GWO interface allows you to look at a list of all 9 combinations of picture and text, or a list of all 5 variants. You might see from the combinations page that the page showing 'hat me up' and the model together is better than the original by a small margin. There's nothing very compelling there though. Naturally, you are tempted to look further. The variants tab shows that 'hat me up' is actually a little worse than the original. What conclusions can you draw about the 'hat me up' text from this?

• it would be racing ahead if it weren't held back by the two 'hat-only'
  pictures.


• it is holding the model picture back. That's why the winning combination isn't beating the original by as much as you expected.


• it is probably somewhere between a little better and a little worse than the original.

The truth is that drawing any conclusions at all from the extra information you gained from the variants tab is not really worthwhile. You could persuade someone of any of the conclusions using the data you have - but none of them are anywhere near as useful or valid as the data you had to start with: the combination of the model's picture and the 'hat me up' text is probably a little better than the original.

One of the Google team explained yesterday that the 'fractional factorial' view (the list of variants) is provided for those that want to use it, but the 'full factorial' view is the one that they really recommend. Certainly, in our experience, we've never seen a case where studying the list of variants led to a decent actionable insight.

What is a Google Authorised Consultant?

Google provide Google Analytics and Google Website Optimiser for free. It makes sense for them to do that, because these two applications make websites better for users, which makes advertising through AdWords more valuable, which makes Google money.

However, it doesn't make sense for them to provide in-depth support for these applications. To do it right, they'd have to engage deeply with each client, understand their business's specific needs and challenges, and maintain a relationship with each client over time - and all this over dozens of countries and time zones.

Instead, what Google do is to certify and authorise private companies all over the world to provide support for these products on their behalf. As well as enabling all the aspects of support described above, this allows there to be specialists in different industries and types of website, and support companies of all different sizes, so that users of Google Analytics and Google Website Optimiser can find the right fit for their business.

How the Obama campaign used Google Website Optimiser to win the election

Dan Siroker gave a fascinating keynote speech at the Website Optimiser day at the Authorised Consultants summit yesterday. One of the stories he told of his time on the Obama presidential campaign really brought home to me the power of testing - and the importance of remembering not to trust your instincts!

The landing page of barackobama.com was a pretty simple proposition - a large image or video, and a call to action underneath, which would lead you to sign up to be emailed by the team and asked to donate to the campaign.

Dan asked the audience of web optimisation professionals to guess which photo or video, and which button text, had led to the most sign-ups. There was some variation, but all in all we pretty much plumped for an inspirational video, and a button that said 'join us now', or something similarly imperative.

First, Dan revealed that these were pretty much what the campaign team had expected to do best too. Then he revealed that in fact, the best converting page had had a picture of the Obama family together, and a button that said 'learn more'.

It wasn't what any of us would have thought - but then, if we could tell what was going to work, we wouldn't need testing, would we?