<mark>How to create a backlog of experimentation ideas?</mark>

How to create a backlog of experimentation ideas?

A well-designed experimentation system allows a company to accelerate growth by creating faster feedback loops and enabling incremental delivery.

It allows for systematic testing of a diverse set of ideas from across the organization and ensures that the lessons learned are internalized by all. In addition, it reduces the risks associated with large product releases by allowing the team to start with small versions and validate the opportunity and risk before expanding to the entire user base.

To have a successful experimentation program, product managers need 3 things:

  • 1. A powerful experimentation roadmap with a large backlog of ideas prioritized by value.
  • 2. A cross-functional alignment with the rest of the company.
  • 3. The right data and tools for experimentation.

A powerful experimentation roadmap with a big backlog of ideas prioritized by value.

First, let’s make sure you have a large pool of ideas to draw from.

Allocate time regularly – Group brainstorming session with a pre-set agenda or allowing each team member to reflect on their own experiences

Make lists – Lists of things others are doing (AsSeenOn Session), lists of things we would like to do, and lists of things we need to improve. Reviewing the lists regularly allows us to evolve or blend ideas over time to create very successful experiences.

Involve everyone – The most original ideas often come from team members who are least aware of existing biases and limitations. Brainstorming and sessions with the entire company helped diversify test ideas and even raised strategic questions that forced us to reevaluate our assumptions.

Put a process in place – Cross-functional ideation doesn’t just happen, it needs a process! Soliciting ideas from a wider audience requires constant communication, reminders, and even incentives.

Make it accessible – The easier it is for your team to record their ideas as they come in, the more ideas you will have. Make the idea repository available to everyone, they might even be inspired by reading the ideas that others have come up with.

Give credit – There’s nothing worse than having a great idea that generates a ton of value and not being recognized for it. Even if the experiment is based on the evolution of a few ideas, make sure the people who contributed to those ideas get the credit they deserve.

Start with a real problem – To get valuable ideas, we always try to start with a problem our users are facing. It’s very easy to fall into the trap of ideas that are based on personal preferences and are probably not relevant to our customers.

Test everything! – Sometimes teams ignore ideas because they think “I would never do that” and our customers probably won’t respond either. we basically believe that everyone thinks and acts like us, and that our preferences will apply to our users. As smart as we all like to think we are, we don’t always know how users will behave and the less obvious ideas are often the ones that set companies apart.


Focus on the goal of the journey – Don’t try to fix the retention problem, look at the small steps that improve retention and come up with ideas for those.

Listen to your users – Whether it’s through direct customer research or user feedback on your product, you can get a ton of ideas that customers actually care about. Your customer support team is also a great source of pain points and suggestions for customers.

Get inspiration from others – Talk to other teams who are facing similar issues or challenges and ask them what ideas they are testing and why. We also regularly look at other companies that are not in our market but are facing similar challenges and have a similar customer base. We then try to understand why they are doing certain things and what parallels we can draw in our business to test them.

Be comprehensive with each idea – Whether it’s the number of ways to test a hypothesis or the number of variations of each hypothesis, make sure you cover as many things as possible to maximize results and learning.


Now prioritize all of these ideas:

Create a prioritization process that is easily understood by everyone, but leaves room for ambiguity and change. Once stakeholders agree with the prioritization process, it will be easier to accept the prioritization in progress.


Create consistent criteria for measuring the impact and cost of experiments so that they can be easily compared to each other. Use frameworks, which essentially assess the impact, certainty of results, and effort for each experiment and create a combined score. This combined score is then ranked to create the basis for the priority list.


Remember, prioritization is an art and a science. It is important to balance simple, low-risk experiments with those that are uncertain but have the potential to move the needle.


By definition, your estimates of impact and certainty will be inaccurate, especially when you are just starting out in a given field. However, this should not deter PMs from doing their best to create a relative impact scale for their prioritization score.
Begin discussions about prioritization and alignment as early as possible, but don’t commit to a priority too far in advance. As you learn from testing and the business evolves, you should strive to be agile and constantly reconsider your priorities.

Involve all stakeholders in setting priorities, let them see the process, the tradeoffs that are made, and how changes to the plan are handled.
Have a living document with priorities, but do not expect stakeholders to constantly refer to it. Instead, plan a cadence for communicating upcoming priorities and any changes in a consistent and predictable manner.


Find a way to make the opportunity cost of each experience tangible to your team and stakeholders. This can be as simple as talking about the number of development hours or user traffic that will need to be spent on that experience. It’s good to have a medium-term forecast of the resources needed while maintaining an ongoing prioritization of how to adopt new changes.


Don’t forget about technology debt and bugs. Even if you focus on high-speed testing, you need to maintain your systems and platform to keep testing for the long term.
Keep a backlog for ideas that are not on the priority list. It may not be the right time for that idea, but later on, it could become your top priority.

* indicates required

Intuit Mailchimp

Write first comment

Leave a Reply

Your email address will not be published. Required fields are marked *