Tying experiments to outcomes

Before we get into the canvas itself, I want to talk briefly about how experiments tie to the outcomes we are trying to achieve and the problems we are trying to solve (Know the problem you are solving).

Every experiment begins with a problem to solve. A problem to solve has one or more experiments associated with it. Each experiment, however, is associated with only one problem to solve.

Let’s take a look at how this all ties together:

We start with a desired business outcome and identify what we believe to be the current barriers to that outcome. We then look at possible solutions to each of those problems. By the time we get here, we have assumptions about possible solutions teetering atop our assumptions about the possible problem to solve which are resting precariously on the assumption that this outcome is accurate and worth pursuing. That’s a lot of assumptions. With that much uncertainty, it is a good idea to take an experimental approach. Try a thing, see what happens, and be ready to persist, desist, or pivot.

Experiment - you keep using that word

I first gave the talk “The Experimentation Mindset” back in 2013. In the ten years since, I have often wondered if using the word “experiment” is making it easier or more difficult for our industry to adopt the mindset of continuous feedback, double-loop learning, and empiricism.

What I see, which is common with us folk, is that we get into debates over what exactly is an experiment and how do you do it the most correctly. I think that somewhere between strict empirical adherence and casual negligence, there is a wide zone of rigorous sufficiency where most of us would be just fine.

This canvas comes from that zone; not too strict, not too casual.

The Canvas

The canvas is broken up into four different sections, each intended to focus on a particular aspect of the experiment - The Observations & Hypothesis, Experiment Structure, Learning Loop, and Additional Details.

I am often asked if it would be okay to recreate the canvas in a Word or Google document so that you could add a lot more content to it. In my opinion, that’s not a great idea. The canvas is intentionally space constrained. The idea here is to create something succinct and focused that culls away the noise and gets to the essence of the experiment. If you need a lot of space to explain the experiment, it is probably more that you haven’t given it sufficient thought than it is that what you are doing is so incredibly complicated.

Observations & Hypothesis

The first section of the canvas sets the stage. 

Background

Here we record the outcome, problem, and potential solution, along with any supporting observations. We may want to identify the potential business value that could come from this experiment and tie it back to the potential business value from the Problem/Opportunity. It is very unlikely that this one experiment will unlock all of the value potential in the Problem/Opportunity. Put some thought into just how much of the value you think this really could unlock. Doing so will help you prioritize multiple experiments against each other.

Hypothesis

Our hypothesis is a simple statement that summarizes the experiment. “We believe that doing [action/countermeasure] for [these people / this process] will achieve [these outcome(s)].”

If your “doing” or “for” contain conjunctions - “doing this and that or the other for groups a and b” - your experiment is probably too broad. Find ways to narrow it down. Consider splitting the experiment at the conjunctions - focus on one action for one persona.

Experiment Structure

On the second page, succinctly record the essential elements of the experiment structure.

Consider if there will be more than one variant, what the sample size will be for each variant, and criteria for expanding or ending the experiment. Consider the timeline required to sufficiently execute the experiment. For example, while it may be tempting to run a multivariate experiment in order to get to a conclusion sooner, if you do not have sufficient traffic, that approach may take you longer to get to statistical significance and make a decision that if you simply run a series of a/b experiments with your most promising experiments first.

Along those lines, be clear about what you are measuring for the experiment and how the measurements align with the potential solution, identified problem, and desired outcome (Validate before, during, and after). Pay attention to possible side effects. For example, if your goal were to reduce cycle time by 20%, would it also make sense to measure quality and check for any queuing in downline processes? It certainly might.

Learning Loop

On the third page, there are two sections; what happened & what now. Well, I gave them different names than that, but you get the idea.

Outcomes

This section gets filled out at the end of the experiment. How did our measurements move? What were they? Do we consider this experiment to have moved us toward our desired outcomes (progressive), had no real impact (neutral), or moved us away from our desired outcomes (regressive)?

Next Steps

The “Next Steps” section should be filled out before we even start the experiment, but certainly before the experiment concludes. Here we identify what will happen based on our results. I find it best to think this through up front and make some key decisions before the experiment starts. There is something strange about the sunk cost fallacy or perhaps loss aversion that makes it hard for folks to rollback and delete an experiment once it is complete unless they’d previously agreed to that action. This section does not need to be comprehensive at the onset, but it should speak sufficiently to each potential outcome.

For example, you can agree that a progressive or neutral experiment should be followed up with one of several other experiments, but defer the specific decision until you have completed the experiment in question.

Additional Details

The final page is optional, but I have found it helpful in keeping the first three pages focused and succinct. The “Additional Details” page is where we place links to any more detailed supporting information such as our complete Opportunity Solution Tree, market research, empathy maps, related experiments, or related metrics.

For More

If you would like access to the downloadable version of the Experiment Canvas, head on over to my substack and become a paid member. You’ll get earlier access to all content along with access to exclusive content for paid members.