Book Review: Experiment! by Colin McFarland

Book Review: Experiment! by Colin McFarland

Share these actual insights

“You are not alone.” Okay, it sounds kind of corny, but this is what I felt when I was reading the book. When I was reading the first few pages, I was taken aback about the honest and open approach on website conversion optimization.

Unlike many other popular books on the subject, Colin’s book doesn’t beat around the bush and gets straight to business. Counting 10 chapters (effective 8) and 149 pages, this book will be for anyone who quickly wants to dive into website conversion rate optimization, or refreshen their view on it.

Colin McFarland in more than 31 words

Colin McFarland comes across as a humble and pragmatic person. In his book, Colin describes himself in no more than 31 words. No intro, no history, Colin has made sure to stick to the topic throughout the book, which is why I love the book… no beating around the bush. What can I tell you about Colin that he hasn’t shared with us in the book.

Currently, Colin works as AB/MVT Conversion Manager at one of Europe’s biggest retailers headquartered in the UK. Along with his work as chief experimenter, Colin has also released a text editing app for iOS under the name Pop. Now that he has added ‘author’ to his list of skills, I am starting to wonder if there is anything that Colin can’t do… pole vaulting jumps to mind.

Talk the talk… ?

In any case, Colin has proven himself time and time again to not only be able to talk the talk, but also walk the walk. His book, ‘Experiment!’, is his attempt, and a good one at that, to share with us his experiences with getting things done.

If you are experienced, or just beginning in conversion rate optimization through A/B and multivariate testing, this book is a must have for any avid reader on the subject of CRO. For me personally, it was a great refresher course. To put it a little poetically… a breath of fresh air that blew away the cobwebs in my head which resulted in the regeneration of my faith and motivation in getting started with experimenting (I had recently been doing a lot of work and focusing on research).

The book, don’t forget to write about the book!

Accept Failure… now people will probably call me a pessimist for saying that this is probably my favorite paragraph in the book. Colin rightfully talks about experiment failure immediately in chapter 1.

“Considering the iterative nature of experiments and discovery, the idea of winning the first time and every time in experimenting is a ridiculous one.”

Colin’s comment, although not groundbreaking in the least, is often the most overlooked truths of experimenting. In most companies, business case upon business case must be made just to even start a single experiment, or even make a change (without experimenting… blasphemy!!!). Research, quantitative and qualitative will give you facts, but like anything else, it is not the real world.

Ronny Kohavi of, quoted in Colin’s book, states:

“Google ran approximately 12.000 experiments in 2009, with only about 10% of these leading to business changes.”

From Approach to Results

The book ‘Experiment!’ is split up into 3 parts, Getting Started, Interlude, and Moving Forward.

Getting Started

The first part, Getting Started, discusses the practical side of experimenting. Colin shares his thoughts on the basic stuff that needs attending before anyone should even start with an actual experiment.

  1. How do you create a test plan?
  2. How to control your experiments?
  3. Which method should I use?

A well-known fact is that experimenting requires a prediction. What do you expect to happen? Colin helps us focus on where to begin, the proverbial low hanging fruits, or as he says ‘start with the obvious’.

Colin doesn’t spend too much time on tool selection, which I find good, because tools will only help you to perform the experiment and assist you in gaining insights.

‘Be brave’ Colin tells us a little further in the book. Although we need to be responsible when we experiment, we also need to be brave. Forget testing the water, you need to dive in. Even though many people can own testing, and the whole company needs to be attuned to optimizing (well, if you want to make more money and improve the experience), Colin highlights that a company needs champions. If you read this book, you could be well on your way!


Part two is a nice break from the overall practical approach of the book. Colin, in the second part, talks about 3 topics:

  • User Experience
  • Design
  • Ideas

The UX topic dives into looking at the big picture when considering an experiment. Colin warns for the  wretched ‘Frankensite’. When experimenting make sure you keep the overall UX in the loop to prevent the website’s improvements becoming fragmented, effectively ripping the UX fabric that previously existed.

In Design, Colin’s message is that Design should be done with a purpose. The purpose should always be to benefit the customers/visitors. With a clear message to designers, Colin uses a quote from Ryan Singer to make sure you stay focused on the experiment at hand:

“You can’t improve a design when you’re emotionally attached to previous decisions. Improvements come from flexibility and openness.”

Designers should remain passionate about what they do, but emotions have no place in the experimentation process.

The two biggest influences on design besides emotion are competitor bias and the HiPPO. Colin says we shouldn’t blindly copy from the competitors, but use them as inspiration. As for the HiPPO… we’ve all had to deal with these situations before. Colin recommends that we present experiment plans instead of ideas, since ideas are easier to shoot down by stakeholders and HiPPO’s.

Moving Forward… Conclusion

Almost 900 words to this blog post. You know what… I would love to tell you a lot more about the 3rd part, being an analyst/researcher one of my favorite parts of the book, but I don’t feel like taking all the wind from Colin’s sails. I guess you’ll just have to go and read the book yourself.

My conclusion should in no way be a surprise. This book isn’t the most in-depth book on conversion optimization, and A/B and Multivariate testing, but it is one of the most effective books for you to read. Like the book itself, straight to the point and without too much of that pink fluff and silver lining, Colin’s approach is head on.

Like Colin said, don’t test the water, just dive right in. If you can’t swim in the sense that you are new to conversion rate optimization with A/B and Multivariate testing, don’t worry, consider this book to be your life vest that will help you keep your head above water.

Now, go out there, read the book and become that champion!

[button color=green url= target=blank]Order Experiment! now on[/button]     [button color=blue url= target=blank]Order Experiment! now on[/button]

About Matthew Niederberger

Matthew works as Conversion Optimisation Manager at Ziggo BV. In his free time he enjoys family life as well as digging into online user research material whilst frequently generating some of his own, which he freely shares here on


  1. Vlad says:

    The book treats the a/b testing platform as a black box, which I am not comfortable with. There were a few statements in the book that leave me with questions. For example, Colin advises watching tests constantly (pg. 30) and cutting tests short when the results are much higher than expected (pg 51).

    Both of these practices are contradicted on statistical grounds by (among others). Evan says you should never peak on a test and should set a sample size (i.e. duration) before the test and run the test to the end. And some features in widely used tools are apparently statistically invalid – biased towards letting clients have positive results quickly, be they reliable or not.

    Do you have any info about Colin McFarland’s credentials? I couldn’t find any online during a brief search.

Leave a Reply

FREE Course!

Get FREE access to my User Feedback Fundamentals course at Udemy, a $29 value. Just sign-up to my newsletter.
Unsubscribe at anytime and keep access to the course.

Nope, we don't deal in spam, not even the canned sort.