The testing phase of any web development project is a double-edged sword. On the one hand, it’s really, really exciting but, on the other, it can be a torrid, drawn out affair.

It’s exciting because after a relatively long wait (doing nothing but specing, wireframing, adjusting, planning, scheduling, monitoring etc…) you finally get to see the application working, well… more or less working. It’s the gap between the test version (a bit broken, that’s why it’s a “test” version) and the live version (the version that should be “perfect”) of an app or a website that can become a nightmare for users, clients and developers alike.

With the good, the bad and the ugly side of traditional testing still in effect I have become interested in another form of testing: A/B testing. I’m going to be futile for a second but allow me to explain what A/B is all about…

A/B testing is a bit like going shopping with a group of friends. It’s much, much easier to choose a dress when a bunch of girlfriends are there to help you choose which one to buy. In A/B testing, users are your voting panel, even though they may not be aware that their behaviour is dictating future changes…

Lots of brands have adopted A/B Testing: for example, a famous case is the Google Blue study where the search giant tested which shade of blue users would prefer for hyperlinks. True story! Another example is Facebook removing the “Most Recent” option in their feed. While a large part of Facebook users (including me, yay!) didn’t feel the pain, other users were “chosen” as guinea pigs without notifications and expressed their confusion straight away.

What I quite like with this testing technique is that it follows an empirical method. It is an experiment with no precise hypothesis to guide the process; it’s all strictly trial-and-error.

As advised by Joshua Porter, ideally, changes that are to be A/B tested should be big changes to get maximum benefit. This is why, and every wise man keeps repeating this, even though testing is a love & hate process it’s extremely important to allocate time, money and resources to testing. Testing, and a fortiori A/B testing is an investment for the future of the website.

However, as always, there’s a risk associated with this investment. First of all, it takes time to develop and deploy 2 versions of your website “just” for testing purposes. But also, users may not pick up on the new online goods you’ve just released. This may happen even to the best of us – for instance – the recent announcement from Google that they are stopping the Google Wave adventure.

As explained in this blog, the world wasn’t ready for such a disruptive innovation. Clive Thompson from Wired magazine pondered that the main error Google made was that they relied on early adopters rather than focusing on mass consumers. Maybe they shouldn’t have been so picky about their guinea pigs!!