· 1 min read

The Week in Experimentation #1

This week: Bing’s $100M headline test, how long to run an A/B test, the best free testing tools, the Obama campaign’s splash page, Clarity vs Hotjar, and product strategy.

Every Sunday we round up the week’s posts in one place, with the single idea from each worth keeping. Here’s week one.

This week’s posts

The Bing ad headline test that made $100M a year

An idea rated low priority sat in the backlog for six months, then lifted revenue 12% once someone tested it. Keep this: make tests cheap enough that low-ranked ideas get a chance.

How long should an A/B test run?

Decide the sample size before launch, run whole weeks, and read the result once. Keep this: “until it’s significant” is not a stopping rule.

The best free A/B testing tools in 2026

PostHog, Statsig, GrowthBook and VWO all have usable free tiers. Keep this: check your traffic first; a free tool can’t fix an underpowered test.

How the Obama campaign’s splash page test beat the team’s favourite

Every video lost to every image, and “Learn More” beat “Sign Up”. Keep this: the option everyone loves is the one most worth testing.

Microsoft Clarity vs Hotjar

Clarity is free and unlimited; Hotjar adds surveys and feedback. Keep this: start with Clarity, add Hotjar when you need to ask why.

Product strategy: a practical guide

A strategy is diagnosis, guiding policy and coherent actions, not a list of goals. Keep this: if no stakeholder is disappointed, nothing was chosen.

Try this next week

Pick the idea at the bottom of your backlog that would take less than two days to build, and put it in a test. Use the sample size calculator to set the stop date before you launch.

Share

LinkedInX

One real experiment every morning

Five minutes. Free. Unsubscribe anytime.