10,000 Hours Of Experimentation

Why innovation comes from testing, testing, testing and… more testing.

You might want to re-think sleeping with your blackberry. Now that Malcom Gladwell’s 10,000 hours to mastery is no longer the rule to win.

The better option? Eben Pagan’s trending 10,000 experiments rule.

Experimentation is not only THE buzz word, but the number one strategy for brand leaders looking for industry disrupting results.

“Experimentation is the engine that drives innovation,” according to Roberto Croci, head of Google Analytics 360 Sales, EMEA Emerging Markets. “In this new, data-rich world, a growth-minded culture focused on testing and optimization offers a path for making smart and user-friendly choices that will flow straight to your bottom line.”

Experiments show us, with scientific rigor, what works and what doesn’t work, says Prof. Stefan Thomke of Harvard Business School, and host of “The Thomke Talks: A Five-Part Series on Innovation & Digital Experimentation.

Are You Missing Out On Millions?

In a Harvard Business Review article, Thomke shares the story of a Microsoft employee working on ad serving at search engine, Bing. The developer’s idea languished in a cue of more than a hundred others for six months. Up until an engineer, who saw that the cost of writing the code for it would be small, launched a simple A/B test to assess its impact.

“Experimentation is the engine that drives innovation.

Roberto Croci, head of Google Analytics 360 Sales, EMEA Emerging Markets.

According to Thomke’s article, within hours, the new headline variation was producing abnormally high revenue, an astonishing 12% increase—which on an annual basis would come to more than $100 million in the United States alone—without hurting key user-experience metrics. It was the best revenue-generating idea in Bing’s history, but until the test, its value was under-appreciated.

Test All the Time, But Test Smart

Facebook’s Mark Zuckerberg said in a recent interview: “One of the things I’m most proud of, that is really key to our success, is this testing framework … At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.”

When Facebook runs experiments, it tests new platform features on just a small fraction of users, within a specific market or country. This gives Facebook super control over how it’s able to test, get feedback, and improve customer engagement without disrupting too many people’s experience all at once.

Embrace Failure

Testing, and learning from testing, should become central to any organization’s decision making. Think of it in terms of the principles of the scientific method. They work as well in business as in any other sector of life, says Thomas H. Davenport, who is the president’s distinguished professor in management and information technology at Babson College and a research fellow at the MIT Initiative on the Digital Economy. Says Davenport: “It’s time to replace “I’ll bet” with “I know.'”

How do you stop wondering what will work, and start testing for real results? Deliberate experimentation is all about being brave enough to face failure often, and be willing to get feedback on what doesn’t work enough times, that eventually it becomes a statistical inevitability that you’ll discover what does.

 “Fail often, and learn from those fast failures,” says Croci. He goes on to say that while many people focus on success, it’s the failures that are just as important. So how do we make actually good decisions? Says Croci: “And that’s where the world of experiments come in.”

Enforce a Data-Driven Mindset with Teeth

To infuse your organization with a testing mindset, make testing ubiquitous, and give it teeth.

 Gary Loveman at Harrah’s Entertainment has said that “not using a control group” is sufficient rationale for termination at the company. Jeff Bezos of Amazon, too, reportedly fired a group of web designers for changing the website without testing.

Bezos underscored the importance of experimentation in a letter to employees:

“I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment,” he wrote.

The Canadian bank, Toronto-Dominion, too, has a culture in which managers insist on tests for every major initiative involving customers or branches. The bank’s CEO, Ed Clark, once noted that “although the bank might not be perfect, nobody ever criticizes us for not running the numbers.”

There’s More

Netflix is another great example of experimentation. Building around experimentation is what has helped the streaming company build such an insanely sticky product, says Talia van Evergingen, content manager at Taplytics, an A/B testing platform for apps.

Netflix researchers estimate that if a typical user doesn’t find something to watch in the app within 60-90 seconds, they run the risk of getting bored and moving onto something else. The company fanatically A/B tests everything from the content that a user sees when they open the app, to loading speeds.

As one Netflix blogger wrote, “by following an empirical approach, we ensure that product changes are not driven by the most opinionated and vocal Netflix employees, but instead by actual data, allowing our members themselves to guide us toward the experiences they love.”