Company
Microsoft Bing
Element
Ad headlines on search results
Primary metric
Revenue, with user-experience guardrails
Result
+12% revenue, over $100M a year in the US
In 2012 a Microsoft employee working on Bing had an idea for showing ad headlines differently on the search results page. It was cheap to build, a few days of engineering work. It still sat in the backlog for more than six months, rated low priority among hundreds of other ideas, until an engineer decided to try it because it was so cheap.
When it finally ran as an A/B test, the variant produced revenue so high, within hours, that Bing’s monitoring raised a “too good to be true” alert. It was not a bug. According to Ronny Kohavi and Stefan Thomke, writing in Harvard Business Review, the change increased revenue by 12%, worth more than $100 million a year in the US alone, without hurting key user-experience metrics. They describe it as the best revenue-generating idea in Bing’s history.
What changed
The idea was to change how ad headlines were displayed on the results page. The published account keeps the exact design details brief, and that is fine: the lesson here is not the pixels. It is how the idea was judged before and after it was tested.
Why nobody prioritised it
Program managers looked at hundreds of ideas and ranked this one low. That was not negligence: a small display tweak to ad headlines looked like a small idea. Experienced people are good at many things, but predicting the size of an effect before a test is not one of them. Kohavi and Thomke report that at Google and Bing only about 10% to 20% of experiments produce positive results, and at Microsoft as a whole roughly a third are positive, a third neutral and a third negative.
If most ideas don’t work, and the ones that do are hard to spot in advance, the ranking of a backlog matters less than the cost of testing it.
Why the result could be trusted
Two details make this a good example of experimentation done properly, not just a lucky win.
- The alert. Bing treated an unusually large win as a warning sign first. Surprising results are more often caused by bugs, logging errors or bots than by genius ideas, so they get checked before they get celebrated.
- The guardrails. Revenue was not the only number watched. The change only counted as a win because user-experience metrics did not get worse. A revenue lift that annoyed users would have been a loan against future searches.
What your team can take from it
- Make tests cheap, then test more. If running a test costs days instead of weeks, low-ranked ideas get a chance. Bing ran around 80% of proposed changes as controlled experiments, according to the same article.
- Score ideas by effort as much as by expected impact. Your impact estimates are noisy. Effort estimates are usually more reliable, so cheap ideas deserve a slot even when nobody is excited about them. See our guide to prioritising ideas.
- Treat surprising wins as suspects. Check sample ratios, logging and bot traffic before you share a big number.
- Define guardrails before launch. Write down which metrics must not get worse. Our guide on measuring the success of experiments shows how.
The verdict
Shipped, and the lesson outlived the feature: the cheapest way to find big wins is to lower the cost of testing small ideas.
Source: Ron Kohavi and Stefan Thomke, “The Surprising Power of Online Experiments“, Harvard Business Review, September–October 2017.