Key Points
- If only one team “wins” by doing an experiment, then it isn’t that useful of an experiment from a website product management perspective.
- We should always be swinging for the fences: only do experiments with an eye toward scaling up.
- Experiments should either be unwound or promoted at their conclusion.
Experimentation is one of the reasons to have a regular cycle of website
The Wikipedia definition is useful here: “An experiment is a procedure carried out to verify, refute, or establish the validity of a hypothesis.”
Often the word “experiment” is used casually by a specific team in order to introduce some innovation that they want on their particular part of a larger web presence. But if only one team “wins” by doing an experiment, then it isn’t that useful of an experiment from a website product management perspective (since it is so localized). Also, if a change is only to introduce something new for the sake of seeing sparkly changes then we aren’t really testing a hypothesis.
Remember, experiments in innovation don't always work out. That's ok and normal. The possibility of unwinding an experiment (like taking down a microsite where you were attempting something new) should be considered when it's started.
We need to do three things for strong experiments:
- We have defined what we are testing.
- Implementing the experiment (and deciding whether to do it at all) is part of the normal ongoing change cycle.
- We either unwind experiments that disprove the hypothesis or promote those that are proven.
#1 We have defined what we are testing
Although
#2 Implementing the experiment is part of the normal ongoing change cycle
Some change should be fast and other slow, so obviously some experimentation should be possible immediately. For example, potentially changing the writing style could be done directly by a content team on the next content they publish (although of
#3 We either unwind experiments that disprove the hypothesis or promote those that are proven
Consider the team that wants to try a new method of presenting a data table (let’s say it involves javascript and the standards disallow page-specific javascript). An early step is defining how the experiment might be run. One possibility is simply to open up the text editor to allow javascript and the team
Regardless of the approach, we don’t want to clutter our web presence with a variety of incoherent non-experiments that seemed like a good idea at the time but now just demonstrate an incoherent web presence.
So at the conclusion of the experiment, we could take one of three possible next steps:
- Unwind the experiment, especially if the hypothesis is disproven but potentially even if the hypothesis is proven but you decide not to promote the change. Possible ways of unwinding are: rolling back to the way the content was before the experiment, explicitly labeling it as a test, or deleting it. Remember, experiments in innovation don't always work out. That's ok and normal. The possibility of unwinding an experiment should be considered when it's started.
- Promote the improvement proven in the experiment. The most satisfying next step would be to promote the new change broadly across your web presence. In fact, we shouldn’t even be doing an experiment if we don’t think we would want to do this (exception below).
- Leave the experiment as-is (RARELY DO THIS!). The only reason to leave an experiment in place is if it is really, truly stand-alone, not only in how your organization treats it but also how the site visitor relates to it. So for example, if you have two completely different audiences that never overlap then perhaps you would leave an experiment in place without trying to roll it out wider.
Aside from pure content experiments that can be done within your standards, consider the following when experimenting (this checklist is taken directly from Website Product Management):