Robust Experimentation and Testing | Reasons for Failure!

Since you’re reading a blog on advanced analytics, I’m going to assume that you have been exposed to the magical and amazing awesomeness of experimentation and testing.
It truly is the bee’s knees.
You are likely aware that there are entire sub-cultures (and their attendant Substacks) dedicated to the most granular ideas around experimentation (usually of the landing page optimization variety).  There are fat books to teach you how to experiment (or die!). People have become Gurus hawking it.
The magnificent cherry on this delicious cake: It is super easy to get started. There are really easy to use free tools, or tools that are extremely affordable and also good.
And yet, chances are you really don’t know anyone directly who uses experimentation as a part of their regular business practice.
Wah wah wah waaah.
How is this possible?
It turns out experimentation, even of the simple landing page variety, is insanely difficult for reasons that have nothing to do with the capacity of tools, or the brilliance of the individual or the team sitting behind the tool (you!).
It is everything else:

Company. Processes. Ideas. Creatives. Speed. Insights worth testing. Public relations. HiPPOs. Business complexity. Execution. And more.

Today, from my blood, sweat and tears shed working on the front lines, a set of two reflections:

1. What does a robust experimentation program contain?
2. Why do so many experimentation programs end in disappointing failure?

My hope is that these reflections will inspire a stronger assessment of your company, culture, and people, which will, in turn, trigger corrective steps resulting in a regular, robust, remarkable testing program.
First, a little step back to imagine the bigger picture.

This blog post was originally published as


Digital Attribution's Ladder of Awesomeness: Nine Critical Steps

Culture is a stronger determinant of success with data than anything else. Including data.
[People + Process + Structure] > [Data + Technology]
It seems hard to believe. Yet, it is so fantastically true. At least for now. At least until AGI takes over.
Why is this formula material?
The first part of the equation, for better or for worse, improves in an evolutionary manner. The second part of the equation most frequently improves in a revolutionary manner.
The challenge for Senior Leaders is that revolutions seem a lot more attractive and hence they charge full speed ahead. This results in frustration, derailed careers and a massive amount of money flushed down sad places.
Revolutions in our context, almost always fail. Evolution works. Hence, it is dangerous to overlook the super critical importance of P+P+S.
You want to win big with data, with marketing, with transformative digital yada yada and blah blah, evolve. Do so at the fastest pace you can put in place for transformation of the left-side of the above equation, and use the same pace to evolve the right-side of the above equation.
This will ensure that the people, process and structure will be smart enough to take advantage of the smart and wizbang tech.
Maybe this metaphor will help make this real.
You can’t give a toddler a Harley Davidson motor cycle. The moment your start the motorcycle, the toddler is going to start crying. It is not the mistake of the toddler, she is just a toddler after all. It is not the mistake of the Harley, it is a very cool motorcycle. The mistake is yours.
The toddler needs something to steady her, something she can push, something to exercise her legs to