growth experiments examples

Growth Experimentation: What Is It and How to Conduct One?

Growth is fundamental to every SaaS business: growth experimentation is a way to turbocharge your growth prospects and drive customer success.

In this article, we’re going to explain what a growth experiment is, how to create one, unpack how they’re implemented, and how they can help you create sustainable growth over time.

Let’s get right into it!

• The growth experimentation process is a systematic approach to testing new ideas. It involves developing hypotheses about what will drive growth , designing and running experiments, and analyzing the results.
• Why is growth experimentation valuable? Firstly, it boosts the velocity of data and research-driven decision-making . A culture of experimentation also helps drive an approach focused on your target audience. Finally, utilizing the right growth experimentation framework can help improve conversion (alongside other metrics ) at key touchpoints in your product.
• Let’s break down the process for running a growth experiment. Start with brainstorming ideas for problems to solve.
• Next, prioritize your list of experiments: you won’t be able to tackle everything at once, so optimize based on value, feasibility, and potential impact.
• Once you’ve chosen your top priority experiments, it’s time to design your growth experiment template or framework. Think about your hypothesis, the type of testing to conduct, sample size, experiment duration, and which metrics you’ll track.
• Finally, all you have left to do is actually run your experiment. Don’t forget to carefully analyze your experiment results: it’s that insight that’ll show you tweaks and changes you can make across your product to optimize growth.
• So that’s what it takes to get a growth experiment set up. But how do you ensure your experiments are effective and ultimately successful?
• Dedicated growth teams solely responsible for designing, optimizing, and launching experiments can help turbocharge your growth (rather than expecting busy product managers to do it ‘side of desk’).
• Another principle of success is remembering experimentation isn’t a ‘once and done’ activity. Continuous experiments help shorten the feedback loop and drive success.
• It’s also good practice to clearly segment users, rather than treating them as one homogenous group. It’ll enable you to launch and track more targeted experiments optimized for distinct user groups’ needs.
• To make sure you and your team have confidence in the results of your carefully designed and time-consuming experiments, you should ensure you draw from a statistically significant sample of your user base.
• Finally, make sure you’re documenting and reflecting on each example of a growth experiment you run. By adding a detailed writeup into a tracker document, you’ll be incrementally adding to a valuable source of knowledge – one you can refer back to for years to come.
• Choosing the right tool for the job can help make launching growth experiments far simpler: Userpilot is a powerful, multifaceted product growth tool that can help.

What is growth experimentation?

Growth experimentation is a systematic approach to testing new ideas and strategies to grow a business. It involves developing hypotheses about what will drive growth , designing and running experiments to test those hypotheses, and analyzing the results to make informed decisions about how to move forward.

This careful, data-driven approach helps you make effective decisions – rather than fully committing off the back of an educated guess.

Growth experiments are a key component of any successful growth strategy. By taking a systematic approach to testing and learning, businesses can increase their chances of success in the long term.

Why carry out growth experiments in your company?

First things first, let’s take an in-depth look into why running multiple experiments on growth makes sense.

Enables decision-making backed by a scientific method

Rather than making decisions based on hunches and intuition, you make decisions based on experiment results. Ultimately, it’s a way to give structure to your decisions. You also reduce the inherent risk of making a decision and it failing.

Each experiment represents progress and greater understanding. Over time, new experiments can help you build valuable insights that give you a strong foundation of knowledge to draw from: user behavior , expectations, preferences, what makes them buy-in, engagement with new features, and more.

All of that can help inform future strategies for you and your engineering team to consider when pulling together your product roadmap .

Ensures companies follow a customer-centric approach

Growth experiments are based on resolving customer problems and pain points . This puts the customer in the front seat, helping you form strategies that satisfy their needs.

The better your product meets user needs (i.e., the more customer-centric it is), the more successful it will be.

Experiments help keep thinking fresh by encouraging an ideation mentality. You will generate concepts you might plan, discuss, and implement before you ultimately determine the right course of action.

Optimizes conversion rates at important touchpoints

As a product manager, the analysis of key metrics is crucial.

Growth experiments can focus you and your team relentlessly on rapid learning and problem-solving.

By consistently solving customer problems, you’ll inevitably boost your conversion rate. Let’s say your onboarding needs improvement. A growth experiment might help run several tests on an onboarding process that’ll improve the free-to-paid conversion rate.

What is the growth experimentation process?

Next up, we’re going to make sure you have a complete understanding of the growth experimentation process – one step at a time.

1. Brainstorm the customer problems you want to explore

First things first, you need to start with a series of ideas.

You and your team should brainstorm potential tweaks, changes, and ideas for experiments you want to run. It’s good practice to make sure each idea is focused on impacting a key part of the growth process (i.e., things like acquisition or retention ).

Where do those ideas come from?

You can gather them directly from your customers. Feedback will shine a light on where to focus your efforts (although there’s no harm in widening your research too).

2. Prioritize the many growth experiments you plan to conduct

If everything is a priority, nothing is a priority.

You’ve got to choose which growth experiments to conduct. How you do that is up to you, but typically you’ll want to choose hypotheses based on factors like reach, effort, feasibility, and potential impact.

Prioritization frameworks like value and effort or RICE can help you come to structured decisions on experiments that map to your goals .

3. Create the experiments to test

Now, we go from idea to action: you’ll create detailed experiment plans outlining exactly what you want to test and how you want to test it.

A plan should include:

• Type of testing (i.e., A/B testing or multivariate testing )
• Success criteria (i.e., metrics to measure)
• Sample size

Once you’ve fleshed out these details, you’ll have a much more accurate view of which tests you want to run.

Userpilot’s advanced A/B testing functionality will be available in Q4 of 2023 – one powerful hub to design, build and launch tests.

4. Run experiments on your target audience

Now it’s time to execute the experiments you’ve been carefully planning for your target audience.

Whether that’s making changes to your website, marketing campaign, pricing models, or other areas of your product itself, the key here is to track your results.

Without gathering that data, you’ll struggle to understand how the tweaks you’ve made impact your customers’ behavior.

5. Analyze growth experiments to gather valuable insights

You’ve come up with an insightful idea. You’ve prioritized the experiment you want to run, and your teams have fleshed out the plan. Your experiment’s gone well. What next?

The final step is to analyze your experiment results .

This step will show you which variations and changes you’ve made have had the most tangible impact on your chosen success metrics. The data will show you patterns, trends in behavior, and useful insights that’ll help inform product decisions in the future.

What is an example of a growth experiment for a SaaS company?

We’ve covered the key steps in putting a growth experiment together. Now let’s look at an example you can use as a growth experiment template:

• Experiment : An A/B test looking at introducing a checklist to the onboarding process .
• Step of the pirate funnel : Activation.
• Hypothesis : Visitors are more likely to get activated when provided with a checklist compared to the traditional onboarding process. The assumption is that a checklist will increase the activation rate.
• Running the experiment : Use Userpilot to create an onboarding checklist and then trigger it for 50% of your new users. The other 50% of users will see the existing onboarding flow.
• Duration : Run the experiment for at least two weeks to gather enough data.
• Analysis : The hypothesis is validated if the activation rate is 30% higher using the checklist versus the traditional flow.

Tips for creating a successful growth experimentation framework

You’ve now got a solid idea of what it takes to set up a growth experiment. But how do you make it successful? Let’s find out!

Create a growth team responsible for conducting experiments

For any business that’s achieved lasting success, there’s a common attribute: the leadership usually organizes resources to form a focused growth team.

These might take different forms: in a smaller company, you might just have a specialist product manager or analyst rather than a specialized team.

But the point is that dedicated growth teams (and a growth lead) make sure growth experiments don’t fizzle out. One growth experiment should lead to the next, and the next, to achieve growth and drive success.

Carry out growth experiments continuously

Growth experiments should trigger the continuous generation of ideas and execution.

In a nutshell, if your experimentation business strategy relies on you launching just one growth experiment and expecting that to make the difference, it’ll fail.

You should conduct experiments continuously: a structured plan setting out when experiments should be carried out can help keep the momentum up for you and your business.

Segment your customers before you test them

Your customers aren’t one homogenous group. Most SaaS companies cater to a diverse user base with different needs, preferences, and behaviors. Segmentation allows you to tailor experiments to specific user groups, making it far simpler to uncover insights that are relevant to each segment.

Remember, segmentation also maximizes the potential impact of your experiments!

Choose a sample size with statistical significance

Not all experiments are equal. There’s a difference between a biased experiment chatting to a handful of users and a statistically significant experiment that gives you rock-solid insight.

Statistical significance helps you answer whether the observed differences in outcomes are likely due to the changes you made (in the treatment group) or if they could have occurred randomly.

To choose a statistically significant sample, you need to consider the following factors:

• Population size : The larger the population size, the larger the sample size you need.
• Confidence level : The higher the confidence level, the larger the sample size you need.
• Margin of error : The smaller the margin of error, the larger the sample size you need.

They’ll help you determine an appropriate sample for your experiment.

Develop a growth experimentation tracker document

You don’t want to let knowledge slip by the wayside. Each growth experiment you run should build a knowledge repository.

Create a tracker that helps you to document experiment details. Make sure to include your hypotheses, variations, success criteria, metrics, and perhaps most importantly results: what have you learned?

Hopefully, you now have a comprehensive understanding of exactly what a growth experiment is, the process for putting one together, a handy growth experiment template you can use, and tips to make your experiments a success.

Want to conduct growth experiments code-free? Book a demo call with our team and get started! Check out the banner below for more information.

Leave a comment Cancel reply

Save my name, email, and website in this browser for the next time I comment.

Get The Insights!

The fastest way to learn about Product Growth,Management & Trends.

The coolest way to learn about Product Growth, Management & Trends. Delivered fresh to your inbox, weekly.

The fastest way to learn about Product Growth, Management & Trends.

You might also be interested in ...

How to learn faster to grow faster by ben williams, what is a product qualified lead in saas.

Aazar Ali Shad

What is Product-Led Design? Definition, Examples

Growth Experiments Template

Test strategies for scaling your business through experiments.

Trusted by 65M+ users and leading companies

About the Growth Experiment template

What is a growth experiment.

A growth experiment is a systematic method for testing a strategy to scale your business. Let’s say you want to conduct a marketing campaign, or you want to transition your business from freemium to pay-per-use. These decisions are costly -- and they are often leaps of faith. But growth experiments allow you to test these strategies before fully committing, saving you precious time and money.

How to use the Growth Experiment template

Anyone who has ever seen a diagram of the scientific method at a kid’s science fair is familiar with the design of experiments. Experiments can be time-consuming and resource-draining, especially if you are running several at once. Use the growth experiment template to keep track of your experiments. You might find it especially useful if you are simultaneously testing marketing campaigns, social media campaigns, business growth strategies, advertisements, and more. Regardless of what you are testing, here are a few key steps.

First, form a hypothesis. Many companies, especially startups and small businesses, tend to make marketing or product choices because their competitor is doing it. The hope is that if it worked for them, it will work for you. However, that can be a costly and damaging decision. The better choice is to form a hypothesis based on data about your own customers.

For example, let’s say you would like to offer a pay-per-use option for your music editing app. Your data suggest that your customers are happiest after they have used the product for about a month. In fact, users who enjoy the service for a month are most likely to continue using it for a year. You hypothesize that users will be willing to pay for the service.

Once you have your hypothesis, you can start brainstorming possible strategies for growth. Remember, your hypothesis is just an educated guess about how the world works. You and your team still need to figure out you can leverage that information to grow your business.

Start by using the growth template to organize a brainstorm. Try asking every individual on your team to brainstorm on their own and come to the meeting with ideas. As everyone talks it out, you can start ranking the most creative and actionable ideas. If the team gets lost or stuck, refer back to the hypothesis. It can serve as a guardrail so you stay focused on your goal.

To stick with our example, you might come up with a strategy for monetizing your app. After 30 days, your users can switch to a paid model.

Armed with a strategy, it’s time to do some research. Find out whether similar companies have adopted comparable strategies. Have they worked? What could you do better? What would you change? If the strategy still seems to hold water, take care of the logistics you will need to run the experiment: align with other teams, get buy-in, survey the potential impact on existing customers, and study any technical limitations. If the strategy no longer makes sense, you might want to revisit your hypothesis or hold another brainstorming session.

Now, design the experiment. Your design will vary depending on your hypothesis, the size of your team, the number of customers you might impact, and other factors. In general, you will want to specify: any copy you need to include in the experiment, your treatment group and control group, methods for tracking and measuring the experiment, timeline, and a sample size. Importantly, you will want to decide what success looks like. How will you know that the experiment worked? What predicted outcome do you think will occur?

In our example, you could roll out the 30-day paid option to a certain percentage of your customers. Your data suggested that you need to offer this option to 3% of your customers to achieve statistical significance. If 25% of them switch to the paid model, then your experiment was a success.

Build an experiment sheet. Drawing on the information above, you can build out a sheet on your template to help guide the experiment. Here’s a typical outline:

Our hypothesis is…

To verify this, we will…

We will measure...

If the experiment is a success, we will see…

You can also include a section that amalgamates quantitative and qualitative results.

Run the experiment and perform an analysis. Many experiments unfold over weeks or months. If that is the case, consider meeting weekly or biweekly to keep everyone informed. Update the growth experiment template as you go along.

Once you have your results, meet with your team to discuss them and figure out next steps. See if there are any key learnings or implications. Often, you will find that new experiments arise from these meetings. If that’s the case, then start the process again!

Why do a Growth Experiment?

Conducting a Growth Experiment will help give you an edge over the competition. Instead of taking a leap of faith on marketing or product decisions, you can test your strategies before rolling them out to your entire customer base. Growth experiments offer a systematic, reliable method for informing your planning sessions and scaling your company.

Why are Growth Experiments important?

From startups to enterprise companies, all businesses must deal with uncertainty. But growth experiments take some of the uncertainty out of your high-stakes business decisions. Rather than investing time and resources in an unproven strategy, growth experiments allow you to test hypotheses and grow your business with minimal risk.

When to use the Growth Experiment template

Use the Growth Experiment template any time you’d like to test a business strategy, marketing plan, or any decision that impacts the growth of your company. You may find this especially beneficial when testing multiple strategies at once.

Get started with this template right now.

Works best for:

Diagramming

The BPM (Business Process Management) template is a visual tool for modeling, analyzing, and optimizing business processes. It provides a structured framework for documenting process flows, identifying bottlenecks, and improving efficiency. This template enables organizations to streamline operations, enhance productivity, and drive business performance. By promoting process transparency and agility, the BPM template empowers teams to achieve operational excellence and deliver value to stakeholders.

Agile Transition Plan Template

Agile Methodology, Agile Workflows

An Agile transformation roadmap can help you, your team, and your organization transition from rigid compliance-heavy methods to the more flexible Agile way of doing things incrementally. From requirements to integrations to security, you can map out your organization's moving parts as “swim lanes” that you can then update regularly. Use your roadmap as a way to tell the story of how you see your product growing over a period of time. Get buy-in without overselling and keep your roadmap simple, viable and measurable. By using an Agile transformation roadmap, you can avoid getting bogged down in details and instead invest in big-picture strategic thinking.

A Halloween Retro

Retrospectives, Meetings, Agile Methodology

The Retrospective Halloween template offers a themed approach to retrospectives, perfect for the spooky season. It provides elements for reflecting on past iterations, identifying scary issues, and brainstorming solutions. This template enables teams to have fun while addressing serious topics, fostering creativity and collaboration. By promoting a playful yet productive atmosphere, the Retrospective Halloween empowers teams to tackle challenges, drive improvement, and strengthen team cohesion effectively.

Product Brief Brainstorm Template

Product , Product Management

The Intelligent Product Brief Brainstorm template in Miro is crafted to supercharge your product development process. One standout benefit of this template is its AI-powered capabilities that elevate your brainstorming sessions. Not only does it help in organizing and capturing ideas, but it also provides additional insights and solutions, ensuring a thorough and innovative approach to problem-solving. This intelligent feature significantly cuts down the time spent on synthesizing information, allowing teams to concentrate on refining and implementing the best ideas, ultimately leading to more effective and efficient product development.

Kanban Planner 2024 Week & Month

Kanban Boards, Agile Methodology, Agile Workflows

KANBAN Planner 2024 Week & Month template provides a flexible framework for planning and managing tasks on a weekly and monthly basis. By organizing tasks by priority and deadlines, teams can optimize their workflow and ensure timely delivery of projects. This template fosters transparency and accountability, enabling teams to collaborate effectively and achieve their goals efficiently.

Workflow Template

Project Management, Workflows

The digital world requires collaboration, and better collaboration leads to better results. A workflow is a project management tool that allows you to sketch out the various steps, resources, timeline and roles necessary to complete a project. It can be used on any multi-step project, whether it’s a business process or otherwise, and is ideal for plotting out the tangible actions you’ll need to take to achieve a goal and the order in which you need to complete those actions.

The Growth Mind

🧪 Growth Experiments: A simple framework to build them

+ my notion template 🎁.

👋 Hi there, it’s Pierre-Jean. Welcome to this new edition of The Growth Mind!

Every 2 weeks, I share Growth strategies inspired by the world’s leading scale-ups.

If you’re not already a subscriber, drop your email here to receive the next articles👇

In today’s article, we’ll go through:

📝  A simple framework to build growth experiments

🎁  My “Growth Hacking Process” Notion template as a bonus

If you liked this article, please consider giving it a like 💙 and sharing it with a friend to help me grow The Growth Mind.

In Growth, experimenting at a high tempo is key. It’s how you:

Get learnings on what works, and what’s not.

Succeed , by finding the winning tactics you can deploy at scale.

At BlaBlaCar, the leading carpooling platform where I work, Fail. Learn. Succeed . is one of our 6 company’s principles and probably the one that fits the most the culture of experimenting and testing new things. That’s a principle I particularly love.

But experimenting is not that easy: doing it without a clear structure and process can become messy.

That’s why I share today a framework to do it, which I use for many experiments. So let’s go through it!

📝 A simple Framework to build your Growth experiments

1# hypothesis.

→ We believe that [HYPOTHESIS]

Every experiment should start with a problem to solve (found through qualitative or quantitative data) and a hypothesis to solve it.

This step must be well defined as it’s the basis of the experiment.

So when building the hypothesis, it’s important to:

Describe why you think the experiment is worth giving a shot ( problem identified ).

How you’re going to solve it ( solution ).

How it’s going to impact growth ( outcome expected ).

2# Experiment description & methodology

→ To verify that, we will [EXPERIMENT DESCRIPTION]

In this section, the following elements should be included:

What you want to test → eg. an onboarding funnel step, a landing page, a price…

What is the change → a new messaging, a higher price, a new CTA wording…

How you’re going to test it → A/B test, before after…

Being as precise as possible here is essential, as it’s going to help a lot for the implementation, especially if other people like engineers are involved.

A vague description can lead to misinterpretation, and then mistakes in the implementation, making the experiment's results unexploitable.

3# Success metric

→ We’ll measure the evolution of [SUCCESS METRIC]

Defining a single metric of success is key to measuring the success or failure of an experiment.

When experimenting, trying to impact too many things at the same time is confusing and makes the analysis suggest some bias.

That’s why having a single, clearly identified metric, is the way to go.

The metric of success can either be:

Volumes (eg. number of sign-ups coming from an acquisition channel)

A conversion rate (eg. free-to-paid conversion)

An amount in € (eg. average revenue per user)

4# Success Criteria & Damage Control

→ We are right if [SUCCESS CRITERIA & DAMAGE CONTROL]

The Success Criteria is the minimum uplift you expect on your success metric to consider the experiment as a success. Having an uplift is great, but to consider an experiment as significant, having a minimal uplift threshold is better.

Otherwise, you’ll face situations where you have an uplift but you’re not very confident at scaling the experiment because the change is too small.

The damage control is a metric/part of the user experience you want to make sure you’re not degrading with your experiment.

Sometimes, experiments can win at increasing a success metric, but at the same time can degrade another one. That’s why having damage control helps to have a holistic view of an experiment's impact.

5# Estimated time

→ The experiment will run during [ESTIMATED TIME]

The estimated time includes:

the time to build the experiment;

the duration of the experiment;

and the period of analysis.

It gives a clear overview of the efforts and time needed to fully build and run the experiment.

6# Potential blockers

→ We’ve identified X and Y as [potential blockers]

Is there any blocker to run your experiment? A blocker can be technical, a lack of internal bandwidth, volumes, or budget. In summary, anything preventing you from successfully running or analyzing your experiment.

7# Tools needed

→ We’ll build the experiments with [TOOLS]

The tools you’ll need to build and run your experiments. Especially useful if you need to buy/try new tools for an experiment.

8# Estimated cost

→ The cost of the experiment is [ESTIMATED COST]

How much does your experiment cost? Generally, we only include the “direct” budget here (eg. marketing budget) and not the indirect cost (eg. salary of the team working on the experiment).

But for some experiments where a big effort is needed, it can make sense to fully assess its cost for the company.

9# ICE Score

→ During our prioritization meeting, we rated this experiment with an Ice score of [ICE SCORE].

ICE is a framework used to prioritize growth experiments . It’s an acronym standing for Impact - Confidence - Ease.

To prioritize experiments in their backlog, growth teams rate each criterion on a scale from 1 to 5. The addition of those criteria gives the ICE score.

I = Impact → What’s the expected impact of the experiment on the growth team/company north star metric? 5 = strong impact, 1 = low impact

C = Confidence → What’s the level of confidence in this experiment being a success? 5 = strong confidence, 1 = low confidence

E = Ease → How much effort do we need to put into building and running this experiment? 5 = Easy to implement, 1 = Hard to implement

10# Learnings

→ We’ve learned [LEARNINGS] by running this experiment. (To complete once the Experiment is finished)

Whether the experiment is a success or a failure, documenting and sharing learning is key to progress. That’s what will make the team and the company progress in the long run, by gathering knowledge on what’s working or not.

In the learnings sections, we want to know:

What did we learn thanks to this experiment?

Was it a success or a failure? Why?

Is there anything we’ve discovered that was not expected?

What could we do better to increase the impact of the test?

11# Next steps

→ We now have clear data to [NEXT STEP] this experiment. (To complete once the Experiment is finished)

Depending on the experiment results and learnings, there are different possible next steps:

If the experiment is a failure = Kill it or try it differently.

If there are no significant results = Kill it, continue running it to gather more learnings or change the test methodology.

If the experiment is a success = Scale it or try an improved version to see if reaching a higher impact is possible.

The next steps’ decisions are influenced by the data and learnings gathered, but also by the gut and vision of the team.

🎁 Notion template - Growth Experiments

If you want to duplicate this framework, you can steal my complete “Growth Hacking Process” Notion template , which includes:

An example of an experiment built with the article framework.

A board to track the experiment’s progress.

→ Click on “Duplicate” at the top right of the page to make it yours.

Subscribe to receive the next articles directly in your inbox 👇

That’s all for today’s edition. See you in 2 weeks for the next one 👋

If you’d like to share feedback, feel free to reach out to me by answering this email or commenting directly on the article.

Ready for more?

A Complete Guide to Growth Experimentation in 2024

Look at growth as a mindset, not a playbook.

Growth is a combination of clean data infrastructure, excellent customer experiences, and rapid experimentation. Achieving sustainable growth requires an understanding of all stages of the customer journey, and a commitment to ruthlessly improving & optimising each step.

A growth experimentation framework provides the visibility, prioritisation & insights to do so. Three pillars are critical to its success: culture, process, and skills .

Whilst there are best-practice principles that can be applied across businesses, there isn't an out-of-the-box framework you can copy-paste into your company.

It takes time and effort to centralise & tailor your processes, keep teams aligned and build a culture that values experimentation.

Why experiment?

• Sift through a sea of ideas to prioritise your next moves
• Skyrocket growth while maximising return-on-spend
• Rapidly generate insight into your product & customers
• Get executive buy-in for your product road-map
• Build a culture of high-velocity optimisation

Getting over the first hurdle

Kicking off growth experimentation requires a significant investment into team alignment & coaching.

The usual internal mentalities holding companies back are:

A siloed approach to teams

Sales is purely sales, product is purely product, marketing is purely marketing (and so on). There is little collaboration across disciplines. In this set-up, companies often compete with themselves internally (e.g. internal KPIs), sharing little insight between them.

Fear of failure

‍ Where avoiding poor results is more important than success, teams can find solace in traditional marketing and generic growth experiments. Naturally, these risk-averse choices deliver less meaningful takeaways.

Common pitfalls

Such mentalities often lead to the following:

Businesses often mistake A/B testing on their website for growth experiments

‍ This is not the case majority of the time, as the majority of businesses don’t have sufficient traffic to do micro optimisations in a meaningful timeframe. Instead, they should focus on high-impact experiments.

Businesses think experiments must be successful at all costs

‍ Often, it even leads to the ‘shifting of goal-posts’ mid-campaign in order to show success; this minimises an organisation's opportunity for learning and improvement.

Businesses think they need long planning cycles for one-off campaigns, with high production values

High cost campaigns lead to high production values which lead to higher ad spend to justify the investment—a vicious cycle.

Businesses suffer paralysis by analysis

‍ Afraid of making mistakes, employees pass decisions up the chain, slowing down the speed with which they can be made.

Businesses have insufficient analytics infrastructure to measure test performance

The inability to measure the performance of experiments effectively slows down any experimentation culture.

Companies held back by these hurdles struggle to implement a build-measure-learn cycle that let’s them grow their understanding of customers, markets and trends at high speed. On one end, start-ups don’t have the luxury of relying on institution knowledge that has been created over years or even decades—to survive they need to learn much faster. On the other end, the business landscape for established corporates is changing rapidly—whether it’s marketing channels, the competitive environment, or consumer preferences, they need to stay on-top of these changes.

Three factors are critical to introducing experimentation successfully: Culture, Process, and Skills/Tools. 

Growth culture

A growth culture should exist across all areas of the business—from product and engineering through to sales and customer service. However, the growth team needs to have the authority and capability to prioritize and implement experiments across the business.

Growth is everyone’s job

Growth is not only about customer acquisition, but also (or even mostly) involves customer retention, various stages of activation (e.g. from trial to paying customer to advocate), pricing strategy and of course product development. This means everyone can and should contribute their ideas. Within the growth team itself, all areas of the business should be represented as otherwise you may miss the critical insight that is obvious to one but not the other. The growth team can then prioritise, derive and run experiments from these ideas and coordinate implementation of validated. ‍

Celebrate ‘failure’ as a learning opportunity

Fear of failure and being wrong stifles ideas and leads to teams shifting goalposts to demonstrate success, robbing the team and business of the ability to do better next time. In order to accomplish this, ‘framing’ is important. As long as a growth experiment generates knowledge it did not fail. To encourage this thinking we suggest to categorise experiments as ‘accepted’ and ‘rejected’ upon their conclusion instead of ‘success’ and ‘failure.'

Ask for forgiveness, not permission.

Speed is fundamental as the faster you are able to validate or reject your hypotheses the faster you’ll generate the insight to drive further growth. One of the major barriers in larger organisations is red-tape and fear of making decisions without coverage from ‘above’ (this goes hand-in-hand with fear of failure). This slows down the experimentation process significantly. In order to address this, the growth team should have clearly defined authority—and testing budget without return expectations—within which they can run experiments.

Keep in mind: If you make 100 decisions a day and get only 50% right instead of making 10 decisions and getting 100% right you’re still moving at 5x the speed of your competitor. And this assumes you’re not even learning from those incorrect decisions. ‍

Automation-driven > campaign-driven

We want our efforts to accumulate over time, rather than driving one-off results.

Solve problems. Be creative. Collaborate.

‍ Growth hacking is about solving problems. Each experiment needs to minimise cost—in both time and money—and maximise the insight generated by the experiment. At the same time, identifying high-impact tests (more about this in the process section below) requires creativity and collaboration across team boundaries.

Building culture is a slow process. Buy-in and leading by example from senior management is critical in making it possible at all. At the same time constraining the culture shift to the growth team while the rest of the business continues on as usual can lead to counterproductive friction between teams. 

Growth process overview

There are a number of different schools of thought around team structure and processes. We’ve found the following building blocks to work well. ‍

A full-time growth team

A dedicated team to manage the processes, infrastructure, testing backlog, implementation and analysis. This should cover growth, marketing, product, engineering  and analytics experts. Sales, customer service, operations, and other teams should be included in addition to their primary role. For these ‘part-time growth members,’ it is important they have sufficient time carved out of their primary role to actively contribute to the growth team. ‍

An Experiment board

‍ To outline and track your experiments: those in the backlog, planned, active & completed. We like to use , but this works just as well in , Asana, JIRA or even Google Sheets when you’re starting out. There are some specialised solutions, but these are usually overkill. We’d generally recommend using the same project management solution your teams are already using.

Sprint planning is very helpful in focusing the team on high testing velocity. However, the sprint duration is highly dependent on how quickly you are able to generate a sufficiently large sample for meaningful statistical analysis. Some experiments may have to run through multiple sprints, but your sprint duration should be set to allow the majority of your experiments to complete within a single sprint. At the end of each sprint, the team should come back together for a retrospective to discuss not only the experiment outcome and learnings, but also what improvements can be made to the experiment and implementation processes.

Guiding principles & frameworks for ideation, documentation & leadership.

The meat of the growth experimentation process. Let's deep dive below.

Growth process deep-dive

Part 1 - experimentation ideation.

Experiment ideation is a mix of art & science.

Inspiration should be drawn from:

• Customer data
• Customer feedback
• Previous experiments
• Competitors / Research

These factors will form the basis of our experiment rationale. While ideating, it is crucial to flag limitations around data collection / understanding. E.g. We may have strong opinions, but do we have the data to back it up?

Your approach to ideation should differ, depending on whether you're looking through a product-led or marketing-led lens.

Product-led Ideation

What do our customers love  what is our main benefit.

• Most used features
• Products / features with the best customer feedback
• How can we double down? 
• Where is the opportunity? 
• What can we add, and what can we expect in return? 

Why are our / our customers’ pain-points? How can we improve our product?

• Missing features
• Missing products (growth / tech stack)
• Missing products (service-offering)
• Efficiency bottlenecks (sales / support)
• What do you think is the root cause of the problem? 
• How we can overcome it? What return on investment we can expect from overcoming this?

Marketing / Sales-led Ideation

What elements of the funnel (aaarrr) do we need to improve.

• Who is targeted ‍
• What is going to be created ‍
• Where is the idea implemented ‍
• When in the customer journey 

What are my customer behaviours? For example:

• What features do they use the most?
• What pages do they visit?
• How often do they engage?
• What do they buy? How much do they spend?
• When do they engage (are there time/day or other patterns)?

What are the characteristics of my best customers? For example:

• What source were they acquired from?
• What device do they use?
• What is their demographic background? Where do they live?
• What other products/services do they use?

What events cause people to abandon the product? For example:

• What pages have the highest exit rates?
• Are there bugs preventing certain actions?
• How is the product/service priced relative to competitors?
• What is their customer journey? How much time do they spend before abandoning?

What are our North Star Metrics / important growth metrics?

• Revenue (e.g. ARR, GMV) - The amount of money being generated
• Customer growth (e.g. paid users, marketshare) - The number of users who are paying
• Consumption growth (e.g. messages sent, nights booked) - The intensity of usage of your product
• Engagement growth (e.g. MAU, DAU) - Number of active users
• Growth efficiency (e.g. LTV/CAC, margins, ops efficiency) - Efficiency at which you spend vs make $$
• User Experience (e.g. NPS) - How enjoyable customers find the product experience

Part 2 - Documenting Ideas as Experiments

We divide our experiments into:

• Diagnostic Experiments: Experiments to provide better insight into the product, market, personas & customer behaviours. Often these experiments are run to address limitations we face in ideation phase.
• Incremental Growth Experiments: Narrow experiments to validate an assumption/idea (e.g. around a feature) or experiments focusing on specific conversion elements within the funnel.
• Radical (‘Big Bang’) Experiments: High-risk, high-reward experiments - something that requires extensive research, planning & resource allocation & has impact across the entire funnel.

Diagnostic & Incremental Experimentation

Experiments should be listed in an experiments space in a project management board (e.g. Monday, ClickUp) and defined using RICE methodology. Everyone across the business is encouraged to submit experiments. When adding experiment ideas, follow the 'Documenting Experiments' guidelines below.

Radical Experimentation

Think of these as particularly bold experiments, strategies or products that you hypothesise can have astronomical impacts on our growth. Usually, Big Bangs are inspired by diagnostic or incremental growth experiments..

Time will need to allocated across the quarter, for planning and executing. For product-led Big Bangs, the team should follow a typical MVP process (mock-ups, interviews, build, test, optimise). The objective should be to build a minimum valuable product, not a minimum viable product (our learnings & preparation should give us the confidence to impress users from the get-go).

Documenting Experiments

Name of Experiment = Hypothesis

• Hypothesis: What do you think will happen? A clear, short statement without justification or explanation. At the end of the experiment you’ll need to be able to validate or invalidate this hypothesis. This is what goes in the title of your experiment.
• Description/Rationale: Why do you think this? Provide an explanation of the hypotheses— what is the supporting material that makes you think this hypothesis is correct? Write it out in this description, and attach any supporting material in the comment/update section below.
• Implementation: Describe the quickest and cheapest way to test the hypothesis.
• Success Definition: The KPIs and threshold to be reached before the hypothesis is counted as validated. These KPIs can be relative (e.g. an A/B test), absolute (e.g. we need to achieve a certain CAC to consider this channel effective) & even qualitative (good feedback). Generally - the more measured & scientific we can be the better - but it's important to not feel limited in the early days of testing,
• RICE SCORE: Reach, Impact, Confidence, Effort. Give a rating for these factors 1-10, then average them. Higher scores = better. E.g. higher effort score = easier, lower effort score = harder.

As is the general case with experimentation, RICE methodology is a mix of art & science. Wherever possible, we should default to more quantifiable measures (e.g. confidence should correlate with data vs intuition, experiment should be sufficiently scoped to guide effort score (e.g. time estimates).

Before it enters CLOSED you must add to the experiment:

• Results: The quantitative outcome of the experiment based on the KPIs defined in the acceptance criteria and whether the threshold was reached.
• Learning: A qualitative interpretation of the result, key takeaways and new questions/hypotheses the experiment has resulted in.
• Outcome: Validated, Invalidated or Unclear

Structuring Your Experiment Board

In our experience, no experiment board will ever be the same. Each head of growth & team will have preferences in tooling, structure & layout. Tools like ClickUp are great as there are endless permutations of 'views' you can customise.

Depending on whether you are a young startup looking for product-market-fit, or an established scale-up or corporate looking to optimise & skyrocket growth - you may choose to incorporate additional features. For example, adding the 4-fit framework (Market, Product, Channel, Model) or a variation thereof to your board is a great way to frame a company's early experiments against your core burden-of-proof questions. Furthermore, you may choose to add labels for lifecycle stage, experiment type and so on.

The end goal is to create something extremely clear and user-friendly. Everyone in the team (including new starters) should be able to navigate a clear list of completed experiments to understand what experiments have been completed and whether they are validated or invalidated. A head of growth should be able to easily groom and prioritise a backlog.

Part 3 - Leading a Growth Team

Practical tips.

A head of Growth’s role is primarily project management.

A Sprint should be held weekly or fortnightly. Ideation sessions can be held on an ad-hoc basis.

The growth team should be composed of a growth lead to coordinate the process as well as marketing generalists/specialists, BI/analytics, engineering, sales and customer success. This composition can be adapted to your company setup (standalone pod vs matrix vs mixed).

Completed growth experiments should be documented in a company-wide knowledge base and results shared with the wider business, for example by:

1 - Setting-up a wins distribution list/email newsletter for sharing experiment results and business impact with interested parties (this can be only successful tests or also include failed ones)

2 - Publishing experiment results to company dashboards 

All employees should be encouraged to add ideas to the idea pipeline. The growth lead will vet ideas (only to make sure all required information is included and they are specific enough) and coordinate with the ideator if more information is required.

Before the Growth Meeting

Head of Growth reviews activity:

Experimental velocity vs goal (tempo) Coordinate update of key metrics Gather data about tests that were concluded High-level assessment of last weeks experiments & results Review & update new experiments - prioritise/rank them on the basis of finalised RICE score For product-led initiatives the aim is to divide time b/w doubling down on what users love, and addressing what’s holding them back.

Leading a Growth Meeting

Purpose of the growth meeting is not ideation!

Metrics review and update focus area (15min)

North-star and other key growth metrics Key positive factors Key negative factors Growth focus area (AARRR)

Review last weeks testing activity (10min) ‍

Tempo (vs goal) How many "up next" were not launched—why

Key lessons learned from growth experiments (15min) ‍

Preliminary results for just concluded experiments Conclusive results of finalised experiments and implications for future action

Select growth tests for current cycle (15min) ‍

Members give overview of the ideas they nominated Brief discussions and selection of experiments for next week Each selected experiment will be assigned an appropriate 'owner' Experiments worthy of testing, but not ready for next week will be slated for launch pending further information on the timeline from the impacted teams Selected ideas need to fit the current growth focus

Growth of idea pipeline (5min)

Recognise top contributors

Skills and Tools

Finally, running successful growth experiments requires the ability to measure experiment results accurately. This is best achieved with a solid analytics infrastructure.

Growth teams need to combine a wide array of skill sets, from marketing, copy-writing and psychology to analytics and engineering. These skills don’t need to be combined at expert level in a single individual, but all team members should have a good technical and analytical understanding.

Many growth experiments consist of combining existing systems in novel ways to create value—take for example the early-days Airbnb example of posting their listings on Craigslist. This required ‘hacking’ together a solution that was able to convert Airbnb’s listings into a craigslist post and then post it on the platform. In order to be truly creative, growth teams need to understand what is technically possible—and what 3rd party solutions that are out there.

These golden ticket growth hacks are only possible with a robust growth process, culture and infrastructure. They result from shifting away from the traditional approach, moving away from campaign-driven promotions, and looking towards layered, automated strategies. Throw in some hustle, and at the very least you are sure to achieve fantastic, sustainable, long-term growth. Throw in some out-of-the-box thinking, good timing and some good-luck for measure, you are in the perfect position to pull off your one-in-a-million growth hack.

Happy hacking!

Leave it with us.

Our services, our resources, get in touch.

5 templates for running high-velocity growth experiments

As Arjun Sethi, a partner at Social Capital, writes in a Wired article :

Today there’s an endless stream of great content in the form of blog posts and ebooks about distribution, traction, growth, growth hacking, growth marketing and even growth teams. It’s easier than ever to learn because growth isn’t an art, it’s a science, and the methods are binary. You come up with hypotheses, you test them, you iterate and you prioritize based on learning what’s high impact for your product. Arjun Sethi, partner at Social Capital

When you break it down, growth isn't just about finding wins. It's about building a repeatable process that helps you win over and over again.

We've put together a collection of our favorite templates that you can remix and reuse to create a tailor-made growth process for your company.

1. Basic growth experiments

The most important thing in growth is to be methodical. That means setting goals, making predictions, and analyzing results against your original baseline so you can iterate and improve.

In this basic growth experiment template , the main table gives you a bird's-eye view of all growth experiments in the pipeline:

Flip to the calendar view, and you can see your experiments by their start date:

Finally, another view groups experiments according to outcome, making it easy for you to track the results of all your experiments in a single place:

The ultimate goal early on is to set up something simple and basic that everyone in a company can use to track the experiment pipeline.

2. Growth experiments planning and execution

As you expand your growth operation, it isn't always enough just to be able to plan and track all your experiments in a single place. Experiments get complicated, with different tools and different executions. At this stage, it's crucial to connect the dots between experimentation planning, execution, and communicating the results with your team.

The following template helps you manage each of these critical junctures in the growth process, whether it's brainstorming ideas, implementing them, or analyzing the impact on your funnel.

In the following view, experiments are grouped according to their status in the pipeline, from requiring design, to scheduling, to completion and analysis.

A simple brainstorming form allows anyone on the team to directly input experiment ideas, and categorize them with the appropriate metadata:

When you're experimenting to get more growth , you're often trying to move the needle at a very specific point in the customer journey. That might mean increasing sign-ups through marketing campaigns, or testing your onboarding flow to improve user retention. Categorizing experiments along criteria such as where they impact the funnel, how they're being implemented, and the target outcomes provide you with a more granular view of the data.

For example, if we wanted to look at experiment results according to measurable outcomes, we can simply create a view that groups records according to their funnel impact:

In a glance, you can see which experiments have impacted everything from acquisition to retention.

Running a growth team is heavily variable according to the kind of product you're trying to drive growth for. A good starting point is to build a flexible process that allows you to slice and dice your data according to the criteria you need.

3. Scale a culture of experimentation

When you're initially putting together a growth team, your objective is to find areas in which you can drive growth for your product. As you scale the company, the challenge becomes slightly different: how can you enable different teams to run experiments, iterate, and grow? That's the only way you can scale growth repeatably.

Andrea Burbank, a data scientist at Pinterest, recommends : “The problem you want to solve is scaling yourself—figuring out what you're doing that's helping these experiments be successful. Develop repeatable processes and guidelines and checklists so that you don't have to be that single point of failure.”

We put together this experiment checklist based on one of Andrea's slide decks .

Forcing someone on the marketing or product team to run ideas by the growth team each time they want to set up an experiment puts an unnecessary bottleneck on the process and makes it difficult to continually experiment. At the same time, experiments need to follow a method. Otherwise you run the risk of getting faulty data. Setting up this type of checklist for your company helps scale experimentation across the company—beyond just the growth team.

4. Growth OKRs

Brian Balfour, former VP of Growth at HubSpot, recommends that growth teams set OKRs —Objectives and Key Results—to track growth goals.

• Write down your goal
• Set a time period (30-90 days)
• Assign three key results, or measurable objectives that you can track, i.e., OKR1: Improve activation by 50 percent; OKR2: Improve activation by 2x; OKR3: Improve activation by 10x.

The key is to map out key results by their probability of success . In the above example, your team might have a 90 percent chance of achieving the first key result, a 50 percent chance of achieving the second, and a 10 percent chance of achieving the third result. This helps keep your growth goals realistic while leaving room in the margins to achieve stretch goals.

Here's a basic template you might use to track your growth team's OKRs:

The main table works as a dashboard that you can use to keep track of everyone's OKRs on your team. You can see what quarter it's scheduled for, each key result, and whether it's a team or individual OKR.

This dashboard is linked to another table named “Initiatives”, which works as a checklist for what steps are being taken to achieve each OKR:

In a single base, your team has access to high-level, long-term goals, and can dig in more granularly as needed.

5. User surveys template

To get more growth, the first step is often to get inside the heads of your users and figure out what makes them tick. As Y-Combinator co-founder Paul Graham writes , “startups early on need frequent feedback from their users to tweak what they're doing.” The challenge is getting that feedback at a steady cadence and high frequency.

The following template is a base for managing user surveys and NPS scores. The primary table shows a list of all survey responses for Porchcam, a fictional smart camera product.

The table above shows basic user information for each response, and contains both multiple-choice survey responses as well as text fields for longer-form answers as well as notes.

Survey responses can be gathered through this form:

The base also contains a table specifically for NPS surveys:

An NPS survey basically consists of two parts. The first part asks a user how likely they'd recommend a product to a colleague on a field of 1-10. The second part simply asks them why. NPS is a useful, quick way to collect quantitative and qualitative feedback on your users.

Building blocks for growth

Growing something from nothing is hard precisely because the possibilities can feel limitless. Early on, when you're trying to grow an initial product you're often starting from scratch with everything you do—whether that's running Facebook ads or optimizing your onboarding flow. What allows you to scale this process is filling in the white space and building a foundation that you can turn into repeatable processes. These templates are meant to work as building blocks to do exactly that.

Universe creator spotlight: Robert Palmer

How to add a crm to your event marketing workflow.

• Best Growth Freelancer: 2021, 2022, 2023
• 120+ Clients, 1000+ Exp. and €6.3M Ad Spend
• 25K people/month visits this site
• Featured by TedX, TechCrunch & The Next Web

By Ward van Gasteren

One of the first growth hackers in Europe, Author of ' Growing Happy Clients ', and freelance Growth Consultant to companies like TikTok, Strategyzer, Pepsi, Cisco, Unilever and more .

In this article:

Step-by-step: how to plan and track your growth experiments [2024].

• Last edited: January 30, 2024

Author of ' Growing Happy Clients ' and freelance Growth Consultant to companies like TikTok, Pepsi, Unilever and more .

A proper Growth Process helps you to run more experiments, get more learnings and get more wins. 

My name is Ward van Gasteren and I’ve worked as a Freelance Growth Consultan t for the past 9 years with 80+ clients worldwide, including TikTok, Pepsi and Unilever, but also smaller startups like Zigzag, Klearly and StartMail, to help them run better experiments and manage their growth process.

In this article, I’ll show you step-by-step: – How to keep track of experiments – How to plan growth experiments – And share which best practices are useful and which are useless.

And I’ll share some growth experiment templates to save you some time. 

Let’s start! 🚀

8 Steps of Running Growth Experiments

As you might know, the process consists of two preparation steps plus a cycle of six steps, and I’ll walk you through it step-by-step. 

• Picking the right focus / OMTM
• How to find the real bottleneck
• Create a backlog of growth experiments
• Prioritize ideas (We’ll discuss multiple models)
• Plan your growth experiment correctly
• Start working in week-to-week sprints
• How to analyze your experiment results
• Complete Cycle (Iterate, Stakeholders & How to Document Learnings) 

1. How to pick the right focus / OMTM

Before you start on your growth sprint, you as the Head of Growth / Growth Lead need to prepare everything for the team to dive right in. 

The main thing here is to define the right target to work on right now. It will depend on the following:

• What is your organisation’s focus at the moment? (Because you want to make an impact where it matters!)
• Where do you see the most room for improvement ? Maybe you haven’t touched the retention process at all, so just adding some basic emails/notifications would be very useful!
• What gets your team enthusiastic ? Because if they’re not into it, you need push them way more, versus when it’s something they care about as well. 

For corporate growth teams, you can go very narrow and focus on a specific part of your audience/ICP, with a specific use case and a specific part of your Pirate Funnel. 

For startup growth teams, you want to aim at big swings and probably have a good idea which step of your Pirate Funnel is the biggest bottleneck: 

• If your product isn’t retaining users well , focus on your activation/retention first to get to Product/Market-Fit. 
• If your retention is going well , but you don’t have a steady stream of new customers, focus on Awareness/Acquisition first to find a steady Growth Channel and a tactic that is repeatable.
• After that , you can start working on revenue and referral.

Great – you’ve picked a problem to work on…

This will now be your One Metrics That Matters (OMTM) for probably the next 2 to 4 months to run experiments on.

But there is one very, very important step, before you dive into brainstorming experiments: 

“Why is your problem a problem?”

2. How to find the real bottleneck

If I say that you should improve your Retention, you could probably come up with 100 experiment ideas. But if I’d say that you need to improve your Retention because people forget about us at the start of the next quarter, you have a much better idea of how to hit the nail on its head. 

This understanding of what is stopping your users, doubles or triples your experiment impact from my experience. 

So before you start brainstorming, understand your bottleneck by looking at the following.

Ways to understand underlying problem to your bottleneck

• Ask Support or Sales what they know about customers regarding your focus
• Watch heatmaps & recordings for relevant pages, to see what is happening. 
• Analyze the step-by-step click-through-rates for that step. 
• Call some customers to talk you through that step. 
• Check forums regarding your issue to understand what frustrates users

This is a nice step to involve your team to hypothesize together what the reason is why people are not converting. This way you get more buy-in from your stakeholders and get the team more involved in the problem, they’re already thinking about solutions and you iron out some of your personal assumptions by looking at it with multiple people. 

3. Create a backlog of growth experiments

For Growth Experiments, Quantity definitely leads to Quality

Next up, you want to create a huge backlog of growth ideas to fix the hypothesis from step 2. 

How to organize a 1-hour brainstorm / Sprint-Kickoff session with your Growth Team & stakeholders:

• 5 min (or beforehand): Ask everyone to prepare by reading the data 
• 15 min (or beforehand) : Ask everyone to do some inspiration-research by themselves: look at competitors, look at random other growing companies, read about best practices.
• 10 min : Write down your ideas for yourself only! Aim for 10 per person. It’s not about the quality – the weirder the better, because that’s what can help us to make big swings, think outside the box and make big jumps. By doing it for yourself, you’re avoiding Tunnel Focus and the HIPPO-effect!
• 10 min : Discuss all ideas – everyone just reads out their own ideas (without comments from others!) and everyone can ask questions to elaborate on ideas or add new ideas on the board when they get inspired by others. It’s important to keep it a safe space where everyone dares to be weird and creative, so that you can get the best ideas out, and you can always focus in later on the better ideas without hurting people’s creativity. 
• 10 min : Prioritize all ideas – I’ll fully explain this in the next paragraph. To avoid contamination and endless discussions (“Should this be a 3 or 4?”), everyone should score by themselves and you take the average 
• 10 min : Agree on top 3-5 experiment that you’ll begin with & assign owners. 
• (Optional) 15 min : Plan the chosen experiments with a hypothesis, to-do’s and deadlines – the planning of your experiments I’ll discuss after this.

Common pitfalls when brainstorming experiments:

• Hippo-effect: The Hippo-effect describes the dominance of the Highest-Paid Person’s Opinion (The ‘HiPPO’), where it’s common for a company’s CEO, an agency’s client or a teamboss to dominate a discussion with their ideas, assumptions and worldview, which really limits the rest of the team to feel safe to share all ideas and treated fairly.  To avoid this effect, give everyone their own time to think for themselves and encourage everyone to share their thoughts and ideas. Remember, diversity of thought is the key to getting the most innovative and disruptive ideas that could lead to successful growth experiments. So, be open to all ideas and evaluate them fairly, through a fair prioritization model as we discuss in the next chapter..
• Tunnel focus: The tunnel effect (also known as groupthink) happens when a group of people gets sucked into thinking about one idea, often because it was the first one shared or because of peer pressure to agree with everyone else. This often limits the diversity of opportunities thought of by the group. This is why I let people write down their own ideas without discussion, so that everyone thinks in their own direction first and then give every idea the chance to be mentioned. This way you get more creative ideas and uncover different directions, because sometimes (or actually most times…) you’ll find that the best solution isn’t the first one you thought of.
• Forgetting the real cause . Perhaps the biggest pitfall is the ‘Solution-Bias’, where people dive straight into naming solutions, without fully understanding the underlying problem of the goal that you’re working on. This way you can come up with many many solutions, but you probably won’t really hit the nail on the head. For example, let’s say you’re trying to improve the conversion rate on your website, but you haven’t fully understood the root causes of why visitors are leaving without making a purchase. If you immediately jump into generating solutions, you might come up with ideas like changing the layout, adding more images, or running a discount promotion. However, these solutions may not actually address the root causes of the problem. Perhaps visitors are leaving because they don’t trust your brand or they’re confused about your pricing. By taking the time to fully understand the problem, you can identify these root causes and generate solutions that will actually address them. This way you can be sure to hit your goal straight at the core with the most relevant growth ideas that are most likely to have a real impact.

4. Prioritize ideas (Most common to overcomplicate)

Basically you want to know which experiment is the best to start with. For some that’s the quickest win (so small ideas first), but what if all your big growth wins are just a bit more complicated?

I feel that prioritization is just to split the best ideas from the worst ideas . 

It’s now about 1 vs 2 vs 3. It’s about the top 5 versus the rest. You just need to know which to start with and which to save for later.

Because the scoring is subjective (Everyone just guesses if the Potential is a 6 or a 7…) and if you all agree that your second-highest scoring experiment is the best one to start with for whatever reason, you should just start with that one.

I usually use ICE, because I find it easiest/fastest to score, because the score names are easy to understand, anyone on the team can answer the questions and it’s relatively short, but just to give you a complete overview…

The most common models to score your experiment ideas: 

• ICE (Impact, Confidence, Ease) : Impact = How big would the impact be if this experiment is successful? Confidence = How sure are we that this idea would work/how much supporting evidence do we have, like survey responses, heatmaps or best practices? Ease = How easy is this idea to execute (considering the needed resources, skills and technical challenges)?
• PIE (Potential, Importance, Effort) : Basically the same as ICE, but different order and different wording. So check what language is most common in your organization, so that it blends in smoothly. ‘Importance’ goes more in-depth on how on-point the experiment is with the set goal.
• BRASS by Growth Tribe (Blink, Relevancy, Availability, Scalability) : This model is especially useful when scoring marketing ideas. It works as follows: Blink = Based on gut feeling, how well will this work? Relevancy = How relevant is this idea to our audience and product (e.g. Is our audience on TikTok)? Availability = Is this channel available to us based on our budget, skills and tech? Scalability = If this is successful, will we be able to scale this bigger easily?
• RICE (Reach, Impact, Confidence, Effort) : Again, very similar to ICE and PIE. ‘Reach’ is nice if you’d be running experiments in different places (different pages on your site, or on different channels) to compare them.
• PXL by CXL is a very different scoring models, where an ideas can collect points based on yes-or-no questions, like “Will more than 1000 people see this experiment?”, “Do we have supporting evidence that this will work?” or “Is the change instantly noticeable?”. For every ‘yes’ the idea gets 1 extra point, and the idea with the most points is scored highest. It’s a bit more work, since there are many more questions, but it’s nice that it takes some subjectivity out of scoring.

Anyway, I think the scoring model is just a supporting element, so don’t spend too much time on discussing which one to choose and just pick one and adjust it on the go when needed. Try to keep it simple, since you probably have to score a lot of ideas after each brainstorm.

Now that you’ve found your first growth experiment(s) to start with, let’s see how to best document growth experiments.

5. How to plan your growth experiment

Keep in mind that the main purpose of documenting your growth experiment, is that you can hold yourself accountable during analysis and so that future you and future colleagues know what has been tested, and afterwards what you’ve learned from it, so pick a tool that is easily searchable and understandable for your whole team: Hiding learning in separate documents makes this hard, so make sure that you have one central place for your experiments.

These are the six things to document per Growth Experiment for sure:

1. Name + Description

Clearly describe what you test so that everyone understands it. Keep it short. Describe specifics. Look at it through the eyes of an outsider. Some basic questions:

• What will we change where and how?
• What is our reasoning/what led to this experiment? 

2. Hypothesis

Your hypothesis is your ‘Current Knowledge Statement’. I use the following statement, but there are others as well: 

“Based on … [Observation/Data], We add/remove/change … [Action], Which leads to … [Quantified Result], Because … [Customer Reasoning]”

It needs to describe what you currently belief will happen, why that will happen and how that will be measured. 

When the results are not exactly what you expect: Congrats! You’ve learned something that changes your thinking about your audience/business/industry. In the last step, I’ll show you the 4 ways how you can capitalize on these learnings.

3. (AB-Test) Variants

Not every experiment has to be an AB test, but you should always document clearly what you’re putting out there, what the differences are between the variants and make it visual!

The quote ‘an image is worth a thousand words’ is soooo true for experiments, but especially for AB tests: Either you don’t write it down clearly enough, or you write down too much and people won’t read it. Just add some screenshots to it.

4. Metrics to Track, and Targets

You’re not testing a tactic, you’re testing a story. A story that consists of beliefs, documented in your hypothesis. 

The correctness of that story can’t be measured in just one metric. It always consists of multiple metrics and every metric needs to be in line for the whole story to be true. 

For example: If you test a new title on your landingpage, and as expected your bounce rate goes down and conversion rate goes up, but your revenue from that page goes down or stays flat, then it means that your story is not fully correct.

Maybe you thought it would all work, because that’s what user interviews showed you, but maybe you only spoke to a certain part of your audience. 

This is actually a great learning, because now you know that the new version works better for the interviewed audience, but the original worked better for the rest. If you can understand why by talking to those other users, you might be able to create two landingpages that work best for each audience and you can translate that insight to ads, emails, other pages, etc. 

Set targets as a Success/Fail Criteria

Of course, when you run experiments, you should set goals for yourself. I always recommend to set a success and a fail criterium. 

The Success criterium is easy: it’s a goal to strive for, just as you normally set goals ambitiously. It’s the number that when you reach it, you’re super happy.

The Fail criterium is an extra challenge to make sure that you stay honest to yourself: it’s the number below which your experiment is definitely failed. 

Why is this useful? Take this example: You’re running a new campaign and set the goal at 1000 leads. At 1000 or above, it’s clear that it was a success, but what do you do when you hit 990 leads? It’s still good right… So probably you still decide to go through with it. But what at 900? Or 800? Where is the bottom, below which you accept that it was failed. Quantify this beforehand so that when you get to the analysis you can’t lie yourself into an extra win. 

Important Note on ‘Failing’ Most of the time the best learnings come from when you’re honest about a failed experiment, because apparently you, your colleagues and probably also your competitors (if they did this on their website/in their marketing as well), thought that this experiment would be a great idea for the audience, but apparently your actual customers don’t like this that much. This is a great learning of what you shouldn’t waste money on and a sign that you should continue in the other direction.

5. Planning Details:

5a. experiment owner.

Assign one person as the main responsible person to lead this experiment: to call the shots, to talk to stakeholders and chase team members to do their job.

5b. Tasks & Subtasks

Break the experiment down into actionable tasks.

5c. Start & End Date

When do you plan to start the experiment and when should you have enough data to conclude the experiment. This exercise is also a great sanity-check to make sure that you’re going to launch a test on a place where you’ll get enough data to draw conclusions from. You can use this pre-test signficiance calculator to see how much traffic you would need if you would go for a statistically significant result.

To me, it doesn’t have to be significant, because business isn’t science, but we also need to be sure ahead of time that we aren’t waisting resources.

5d. Visuals & Attachments

Again, add some screenshots from your variants or exports of your data analysis leading up to the experiment, so that future colleagues can read what you did and build forward on your work.

6. Start working in week-to-week sprints

The execution should be the easiest part of this process. There are countless growth marketing tools to save yourself some time.

A time management tip from me: The execution of an experiment should stay under 2 full workdays. If you’re making it bigger than that, you’re probably making it too complicated or too perfect. When planning your experiment, you should assume that this experiment will be a total fail so that you are more aware of how much time you put into it. If you’re doing it well, about 50% of your experiments should be correct to your predictions. The other 50% should be there to show you new learnings on risky bets. 

Your job as Head of Growth / Growth Lead

• Task Management Break the experiments down in easy to start tasks, assign them and define them together so that people know what to do. After that it’s just about checking in, if they’re on track to get it done, if everyone understood the experiment correctly and if nobody is being too perfectionistic. 
• Stakeholders who need to know about the experiment when it goes live. For example, Product/Development should not be surprised of a new element on the page and Support/Sales don’t want to get a call from a customer that they can’t answer.
• Your boss/client, who is financing this project, to show them where the resources are going to. 
• Others who might be interested in the learnings from your experiment. As an exercise, I always ask myself beforehand: “What could department X learn from this experiment if it wins/fails?”. For example, your marketing team could improve their email subject lines, Product can take it into account for their roadmap planning, and Sales might want to dive deeper into the topic of your experiment to better understand the customer’s interest. And how great would it be if they can already think about your topic, before the results come back: together you might come to even better conclusions/actions.
• Oversee the pipeline / experiment velocity The easiest explanation of Growth Hacking might be: More Experiments = More Growth. Obviously, I’m cutting some corners there (good experiment planning, proper analysis and drawing strong learnings from it), but next to that it comes down to how many useful experiments you can run every month. If you run double the experiments, you can get double as many wins and double your impact on the organisation. This is all a product of how many insights you can collect that lead to experiments, how many experiments you can set up, how many you can run at the same it and how many you can analyze and act on. It’s a funnel! 🙌 Haha no, but for real: keep an eye on what part of your process gives the biggest bottleneck and set yourself goals as team lead to run more experiments each month by removing bottleneck or planning more strategically. 

7. How to analyze your experiment results

Analyzing your experiment results can be as complicated as you want. Simply, you look at the results based on your metrics to track and the goals you set. From there you can go three ways, but they all end up on the same place: What did we learn and what are we going to do with it?

Tip: Look at different segments, since sometimes your experiment did work well in one audience, but that result doesn’t show on the total audience.

🟢 Win – Numbers are conclusive & positive

Great – we’ve learned that (some) people really love this: Let’s implement this & see how we can do more of this here or in other places. So for example: If leaning into a certain angle, like ‘We’re cheaper than the competition’ works very well, then: 

• How can we emphasise that even more in a new experiment? 
• How can we communicate this USP on other channels? 
• How can we show our price-benefit in other places? 
• What other experiments do we have on the backlog that become more likely to succeed now that we know this? 

🟡 Flat – Numbers are inconclusive, not positive or negative

Not bad, but not ideal: If you’ve tested re-wording the button CTA three times and it keeps showing no result, then you can only conclude that you’re working on something that your audience doesn’t care about enough to make them behave differently. From here on:

• Have you tested this multiple times in different ways? Then move on and note down that this is not an interesting type of experiment to run for now. 
• If you still believe in this, test it bigger/bolder. Maybe your change was just too small, so try going in the different direction: What if we change the whole page to emphasize this? 
• Look at your backlog which experiments should be scored lower as this area doesn’t seem to have the wanted impact.

🔴 Fail – Numbers are conclusive & negative impact

Best! Because apparently everyone in your team believed this was the right way (since you’ve scored this experiment so highly beforehand), but it turns out that the opposite was true. You’ve saved your company a lot of money potentially by not just implementing this without testing.

Probably your biggest fails, become the biggest wins , because it shows that the current state is very much in the right direction. “Do more of this” they say. Opposed to the direction you and probably your competition were going. From here:

• Lean more into the original. If testimonials from corporates didn’t work, since you serve startups, then implement more startup quotes instead, emphasize your startup-use cases more specifically or emphasize ‘for startups’ in your headings.
• Did we make this change somewhere else and can we revert it? 
• How does this impact the scoring of current experiments? Have you scored other ideas high or low, because of your original assumption? Take some time to go through the scoring of relevant ideas to adjust it where the potential has now changed.

The second part of Analyzing Experiments: Learnings & Insights

Here we dive into all the qualitative things you’ve learned from this experiment: What is our conclusion about user behaviour? What kind of comments did we get from customers during interviews or under posts on social media? Were there certain audiences that reacted differently to the experiment? What are all the things we noticed on screen recordings or heatmaps? Every insight can lead to new or better experiments, and you won’t remember all the small details in a year from now. 

And of course, writing down your learnings, makes sure that your team won’t have to re-run this experiment in the future, because they don’t know that it was ever done or that they can’t find the needed learnings. 

8. Complete cycle

So as mentioned it’s a cycle, but I see most growth teams move on to the next experiment on the backlog as a waterfall planning, but instead try to turn your growth process into a flywheel for yourself, so that every time you run an experiment, the next sprint/experiment will be more effective and that you actually build on top of the experiments before.

Iterate ♻️

Instead of going for the next experiment on your backlog, always try to keep space on your next sprint for an iteration on a previous experiment . Now the knowledge is still fresh and you’re building momentum inside your team and organisation. 

Create new ideas 💡

At the end of the experiment, take some time to think about new places where you can add the winning variant or new ideas that sprung out of the insights from this experiment.

Share learnings with stakeholders 🗣️

Ask yourself per department, how can I translate the learnings from this experiment in different places. Maybe Sales can apply this during calls? Or Design can take this into account the next time they work on things related to this? Or can we look into new partnerships based on these insights? 

Implement/Systematize 📂

Lastly, if something was successful move it to development/product to implement the successful variant. If it’s more of a marketing tactic, you should think about creating a playbook regarding this tactic, so that you can make it repeatable, and discuss it with the marketing team on the best way to apply this tactic systematically. If you’re done testing if it works, you can also double down on it by handing it over to a specialist who can get the most out of this tactic. 

Last tip: document, document, document – this is your internal long-term flywheel. 

If you store all the learnings from this year, your team can be so much more effective and you can go much more in-depth when the obvious ideas are gone. Also, when you or another teammate leaves the team doesn’t fall behind, and new teammates can instantly hit the ground running, because they can catch up on everything in their first week on the job. 

Good luck to you all 🍀 & feel free to reach out to me on LinkedIn for any questions. 

Some nice tools to help you:  Upgrow , as a central place to manage your growth process, including experiments, tasks and learnings. For inspiration for experiment ideas, you can check Growth.Design , Gleam’s Library or Ladder’s Tactic Playbook .Or have a look at this list of Growth Hacking Tools to make the execution easier.

• Set success/fail criteria to stay honest about the experiment’s outcome and learn from failure.Failed experiments lead to the best insights: Lean more into the original variant and re-evaluate scoring on other ideas.Document multiple metrics that align with the overall story you’re testing.Use one central tool that is easily searchable and understandable for your team to document growth experiments.Involve the team in hypothesizing to get buy-in and multiple perspectives.Take enough time to analyze and understand the bottleneck by interviewing customers and collecting data.

See this free Growth Experiment Spreadsheet, or have a look at these: https://growthmethod.com/growth-experiment-template/

2x The Experiments 3x The Learnings

• Keep track of your Experiment, Learnings & Tasks
• Easily work together with all your colleagues
• Customize to fit your process, statusses & focus

Ward van Gasteren is one of the first growth hackers in Europe, author of 'Growing Happy Clients', and freelance Growth Consultant to the fastest-growing startups (TikTok, StartupBootcamp, Catawiki) and biggest Fortune 500 enterprises (Pepsi, Cisco, Unilever)

View table of contents

Need help ask me anything, leave a comment cancel reply.

Save my name, email, and website in this browser for the next time I comment.

P.s. Are we already connected on LinkedIn?

Download growth loop framework.

Download Growth Maturity Model

Download your Growth Experiment Outline Card

Download your Experiment Sheet

Download your Pirate Funnel Canvas

Howie Mann Startup Strategy Subscribe Articles Search About Subscribe

product (27)

finance (25)

management (19)

engineering (14)

marketing (10)

All Articles

Popular Articles

How much is Netflix worth in 2024? $US100bn

Return on Invested Capital (ROIC) - Why focus on it?

How Costco avoids failure? Focus on what sucks

Share Buybacks vs Dividends: Who wins?

How to respond to product feature requests? Ask “When was the last time”

Understanding 6 Key Financial Ratios - Best Buy vs Netflix vs Meta vs Apple

REST APIs Explained - 4 Components

7 Types of Bias - Examples & How to Navigate

Kano Model - 3 Types of Product Features with Examples

Employee Training: Who is the Person Responsible?

Data Table UX: 5 Rules of Thumb

User Interface Design: 5 Rules of Thumb

Cognitive Reflection Test - Examples of Reacting vs Checking

How Much is YouTube Worth? Over US$180bn in 2022

The IKEA Effect - Why people fall in love with their own ideas

How the US Air Force Ditched the "Average" and Saved Lives

How to Run a Growth Experiment - Learn the 4 Steps

Financial Model Formatting - 3 Core Best Practices

Finite State Machines - A Common Tool for Product Owners, Designers & Developers

5 Questions to Ask to Find Product Market Fit - User Interviews

Invert, Always Invert - Find out how to kill pilots

How Much is Alibaba Worth? US$900 billion in 2021

What are the Fiduciary Duties of a Director?

Clean Architecture Node.js - Build a REST API

< All Articles

5 minute read | Feb 25, 2022 product , marketing

Confirmation bias is a psychological tendency where we seek information that validates our existing beliefs as well as avoiding contradictory information. This bias often causes teams to waste resources on initiatives that validate existing views but are ineffective in the real world.

This article explains the 4 key steps of a growth experiment and a practical example of how you can run your own growth experiment quickly and cost effectively:

Imagine we are in the product marketing team of a B2B software startup that provides a tool for capturing customer complaints called Capturo.

1. Hypothesis

Set a hypothesis early. Start by identifying a customer struggle and business outcome of what you are trying to achieve. Form your own explanation as to what is causing the customer struggle and come up with a solution as to what could ease that struggle. User interviews are an effective tool at this stage if you have the time.

It is important at this stage to avoid the pitfall of falling in love with your explanation and solution. Focusing on the struggle will allow you to run a new growth experiment if the current one fails.

Example Hypothesis

Struggle: Our new users are observed to create a lower number of customer complaint forms on the platform than our existing users which is leading to lower product engagement and complaints captured.

Explanation: We suspect two core reasons: 1) New users are unfamiliar with the types of questions to ask to gather complaints; 2) New users lack the time or know-how to digitise their existing questions into our platform.

Solution: Our proposed solution will address the first explanation of new users not knowing what questions to ask. We will research and propose the most frequently asked questions to address customer complaints by service type (e.g. contact centre, restaurant, consultation).

Hypothesis:

“Teaching new users the most commonly asked customer complaint questions relevant to their business will help them create more forms on the platform and gather more customer complaints”

Establish a trigger point to communicate with your customer around their struggle and your proposed solution. The most effective mediums are email and calling customers directly. In-app banners, notifications and user flows are expensive if you do not already have the infrastructure in place. As this is a growth experiment we do not need to worry about solutions that don’t scale yet.

The three most important components of a trigger:

• Audience: identify your target audience. Start with bigger groups, segmenting at this stage will be time consuming and take longer to gather results
• Message: keep consistent with your original hypothesis. There is no need to A/B test on a small scale but avoid message variation overkill
• Timing: identify the best time to contact your users to get their attention. Be sparing for when you trigger again due to lack of response

Example Trigger

Audience: our growth experiment will be to email new users who have created two forms or less in the platform from their first three months of joining

Message: “Real examples of customer survey questions [Dental Practices] are using in Capturo” or “Top 6 customer survey questions [Dental Practices] ask in Capturo”

Timing: Send when admin users are most active on the platform (for example mid-week mornings)

Establish an action that you can measure based on the trigger. This can include opening a link to an article, registering interest via a form, logging in to the platform to perform an activity, emailing or calling back to respond.

Without creating an action it will be difficult to measure the success of your growth experiment.

Example Action

Create a webpage link in the email which takes a user to the top customer survey questions. E.g email link. “View top 6 customer survey question examples”

Measure and track the performance of the trigger and action stages. Start simple and use a spreadsheet to track key results. Free or cheap email tracking plug-ins are an effective way of tracking email open and click-through rates.

Once you feel you have gathered enough insight (can take a few weeks) assess the results and determine if your hypothesis was correct. Within the same experiment you can go back and modify the trigger and action to gather more data if needed.

Example Measure

Create a spreadsheet with columns to track the following: User contact email and id

• Email message sent
• Date and time sent
• Date and time opened email (use email tracking tools)
• Date and time opened link (use email tracking tools)
• Already sent (Y/N)
• Created a new template (Y/N) (check manually each week)

After measuring the performance assess engagement metrics for example:

• Total users contacted
• Average contact count per user
• % email open rate
• % click through rate
• % users creating new template

Happy case If the % click through rate and % users creating a new template is good we have quickly validated that knowing the right questions to ask can help boost form creation. We now have an improved business case for investing more resources in-app to build onboarding tours in the form creation stage.

Sad case If the % open rate is poor after adjusting for message and timing then the solution of our hypothesis is likely incorrect. We can go back to our original hypothesis and form a solution for our second explanation (e.g. already know what questions to ask but don’t have the time to create a new form).

Want more tips?

Get future posts with actionable tips in under 5 minutes and a bonus cheat sheet on '10 Biases Everyone Should Know'.

Your email stays private. No ads ever. Unsubscribe anytime.

You will receive future articles from [email protected]

Download Free PDF

🌙 Contact Howie Mann via LinkedIn or Email

• Browse Mentors
• URL copied!

Growth Experimentation Framework: How to Structure Your Experiments

• Founder at GrowthMentor
• 23 Jun 2021

The success of any growth experiment strongly depends on how well it is set up.

• Are you using a standardized and repeatable framework?
• Are your growth experiments prioritized with respect to other potential experiments that could also drive growth?

Here’s a simplified example of how to set up an experiment:

• Experiment Title: A/B test if a chatbot is a good substitute to a form to drive more registrations
• Step of the Pirate Funnel: Acquisition
• Hypothesis: Our visitors are more willing to provide their personal details to an interactive chatbot rather than to a traditional form since our audience is used to interacting with conversational interfaces in their lives.
• Implementation: We’ll build a chatbot using Chatfuel. This will ask our visitors the same questions as our existing signup form. We’ll use Google Optimize to split the traffic to the signup page as follows: 50% will see the form and 50% will see the chatbot. We’ll track the interaction in the page adding a heatmap (using Hotjar) and recording the sessions (also using Hotjar), and we’ll use Mixpanel to build a funnel showing which questions are harder for the visitors to answer and what is our conversion ratio.
• Duration: 2 weeks (to get a data set of at least 1,000 people).
• Win threshold: The experiment is a success (the hypothesis is validated) if the conversion ratio from unique page view to signup is 5% higher via the chatbot than via the form.

Once this process is internalized, you’ll start thinking about implementing features as an experiment.

As a result you’ll start getting better at feature prioritization based on real data.

If you’re currently running growth experiments, speaking to a growth mentor can come in useful.

You’ll be able to share with your mentor how you are structuring your experiments, what you’re measuring, and how you’ve split responsibilities across your team.

Getting a second perspective on your growth experimentation framework can help you head off potential mistakes down the road, increasing your long-term growth potential.

Latest blog posts

22 About Us Page Examples in SaaS to get inspired by (before creating your own)

If you’re redesigning your website or creating a new one from scratch, and want inspiration from some of the best about us page examples on the web, this is the post for you!

Startup Founder Coaching: Where to Find & Best Practices

If you’re a startup founder, we commend your bravery and creativity. But even though you’re quite effective by yourself, you can’t always go it alone. Startup coaching is perfect for burgeoning entrepreneurs who are looking to build their businesses more effectively.

Passive versus Active Learning: What’s the Difference and Why Does it Matter?

Most of us learn passively when we should in fact be taking a more active role in acquiring knowledge. Here is our take on the most effective way to become a lifelong learner.

Get personalized advice you'd never get from reading blogs

• All mentors are vetted for hard skills, but also soft skills like empathy.
• Use dynamic filters to find the perfect mentor for you.
• Find support in a community of like-minded peers.

How To Run A Growth Experiment

You’ll never meet a marketer who says they don’t want to be ‘data-driven’. Likewise, the concept of experimentation—rapidly testing an idea with strict expectations of success and failure—has overtaken the Don Draper days of marketing campaigns and creative ideas. Revenue leaders will usually prefer a growth team over a marketing department, but that’s easier said than done.

Growth experimentation is crucial for the modern marketer, but its success is often much more dependent on the methodology used than the experiment itself. A solid growth experiment doesn’t just drive success for the company, it also makes it much easier for new hires to understand what has been done in the past. A true ‘growth lab’ will show a track record for everything that has been done at the company, and make building on those successes much easier.

In this article, we will break down the steps needed to run a successful growth experiment using the experiment we recently ran at Divisional as an example. This will help you not only understand the theoretical aspects of designing a growth experiment, but will also demonstrate how to implement those steps in real-life scenarios.

Product Marketing: What are you testing?

A common misconception is that growth experiments always involve testing channels. This can be true, but more often, it involves testing a specific assumption within a channel. For example, many startups (ourselves included) want to test outbound email. We can assume:

“Startups with Seed funding but no marketer support have the greatest need for fractional growth help”

If we shortlist 100 startups and reach out to them via email, it’s possible that 0 will respond positively. That doesn’t mean that outbound email is a bad channel, just that our assumption is invalid , and/or our assumption is invalid via outbound email. Failing to note the difference leads to startups quickly invalidating fruitful channels, which can stunt your growth significantly.

The basis of growth experimentation comes from a strong understanding of the product and customer. Every channel can fail for your company, if it’s approached with the wrong target audience and/or messaging.

Before creating a growth experiment, start by outlining all your assumptions and hypotheses about your customers. These can be fairly widespread, where the assumption is about how you believe your target customer acts or thinks, and the hypothesis is how your product addresses this. Some combinations will be stronger than others, so you should prioritize them accordingly.

If you skip this step, then it becomes exponentially harder to identify the reason behind the success (or failure) of a given experiment. The earlier you start building assumptions, the better, as it can also be helpful to reflect on them to see how your thinking has evolved. Founders rarely see their product/customers the same ~ 5 years after starting, and marketers are no different.

How to create a growth experiment

Once you know which assumptions you’re testing, you can move to thinking about the channels behind the test. From the example above, if the assumption is that founders struggle with recruiting marketers due to lack of available talent, and your hypothesis is that showing them a pool of qualified marketers would solve that issue, you can start to ask yourself several questions:

• Where do the founders I’m targeting exist?
• Where would they be most receptive to recruiting material?
• What has worked in the past for us to get in front of founders?

Although these questions are specific to the hypothetical example we outlined, you can make them more generic and widely applicable to any growth experiment. Here's some guidelines to help you get started:

• Where do the [target personas] hang out/spend their time?
• What channels do they typically visit to consume [topic]-related information?
• Which platforms have worked well for us in the past when trying to get in front of [target personas]?
• How much of a lift is it to run an experiment on this channel? Do we have the resources to pull that off?

Choosing the channel

You might assume that founders are most often on LinkedIn when recruiting, and it’s also the easiest place to create a target audience (either custom upload or companies/job titles).

The other thing to consider is how hard it is to launch the experiment (’the lift’). It might be easier for you to build copy and creative for LinkedIn Ads than to scrape a target list and build a copy for outbound email. Therefore, you might prioritize LinkedIn Ads as the channel to reach founders for this experiment.

If it’s helpful, you can also list out alternative channels and reflect on your rationale. This can be useful in the future if you’re looking to test new mediums, or combine them with the one that you end up choosing.

Before you explore a particular experiment, you can benefit from putting your rationale through a test called the ICE Score . It stands for Impact, Confidence, and Ease. The ICE Score forces you to evaluate the tradeoff between how much growth the experiment will drive (impact), how sure you are that it’ll work (confidence), and how many resources it’ll take to get the experiment live (ease).

Creating parameters

With a channel in mind, you can then start creating parameters around your experiment. This will include the expected results and milestones along the way. Statistical significance (stat sig) is an important concept in experimentation, and will inform how you think about your experiment’s expected results.

Stat sig means that your test has sufficient data to be deemed a valid test. For example, if you get 1,000 new users a week, a test that is run on 10 users is likely to be deemed statistically insignificant . Not all experiments need to be stat sig, but determining the goal beforehand improves the integrity of your experiment.

A directional experiment is one that will never reach stat sig, but will give you confidence whether your assumption was correct. For early stage startups, where budgets are tight and there isn’t enough data to get statistically significant results, directional experiments are commonplace. This goes double for startups who are experimenting on their customers (i.e. not with net new audiences), such as Conversion Rate Optimization (CRO) on the website or product. If you are only getting 50 visits a day, then testing a landing page and expecting stat sig will be near impossible.

In contrast, a conclusive experiment is one that is dependent on reaching stat sig. You must make this distinction early on and BEFORE starting your experiment, otherwise you will jeopardize the validity of the results.

Finally, you should decide on milestones to accompany your expected results. This tells you when to check on your experiment and when to make a decision based on the initial results.

See below for an experiment template that includes the parameters discussed above:

Sample growth experiment

Let’s take the example above to create an actual growth experiment, using the template. For starters, we’ll begin with the assumption and hypothesis:

• Assumption: Founders struggle with hiring marketers because the quality of talent isn’t high enough.
• Hypothesis: Presenting founders with a pool of qualified marketers will increase the likelihood that they engage.

Next, we’ll work on prioritizing channels based on the combination that we are testing:

• LinkedIn Ads: Easiest to reach our target audience via custom upload or job title targeting.
• Outbound Email: Lowest cost, but likely that founders are inundated with sales emails, therefore won’t respond.
• Paid Search: Founders are likely not using Search when looking for qualified marketers.
• Other Paid Social: Diverse channels (i.e. Google Display) will yield a lower Cost Per Click (CPC) to reaching Founders, but have much lower intent in their recruiting journey.
• Influencer: Founders would have higher affinity with a recruiting influencer but it would take a larger lift to test (i.e. need a stronger value prop) to incentivize the influencer. Good idea for a follow-on campaign.

With LinkedIn Ads as our chosen channel, we’ll move on to creating the growth experiment:

Rationale: CPC of ~ $9 to target a founder audience (see below), budget of $900. Daily budget of $100 would yield results in 9 days. If typical lead gen conversion rate is 5%, and we’re assuming a 3% variance (2% to 8% CR), then we’d need 894 clicks for stat sig. Given that this is our first experiment, we will make this a directional instead of a conclusive test.

Use this tool to determine the statistical significance of your experiment.

Expected results: Anticipating 100 clicks and 7 qualified conversions (7% CR) for a Cost Per Qualified Lead (CPQL) or ~ $129.

Milestones: The following will tell us, at a budget of $90 per day, how the experiment is going:

• In 2 days ($180 spent, 20 clicks), clickthrough rate (CTR) should be > 0.5%
• If > 0.5%, do nothing
• If 0.3% to 0.5%, pause lowest performing ads
• If <0.3%, begin developing new ad creative/messaging to test
• After 5 days ($450 spent), we should have 2-3 conversions (5% CR)
• If ≥ 2-3 conversions, do nothing
• If 1 conversion AND CPC is higher than $9, increase budget by 20% ($108 per day)
• If 1 conversion and CPC is ≤ $9, do nothing
• If no conversions and CPC is ≤ $9, pause campaign
• If no conversions and CPC is higher than $9, let campaign run to 50 clicks

By setting these milestones, we’re avoiding the following problems:

• We wasted budget by running an experiment that was far off from being successful
• Milestone: Budget vs # of Conversions
• We had incorrect inputs (i.e. proper messaging/creative) that would harm whether the channel is effective
• Milestone: Budget vs Clickthrough Rate on ads
• We didn’t budget enough for the experiment to gather enough data in the given time frame
• Milestone: Budget vs Cost Per Click

Creating growth sprints

An effective growth team will operate using growth sprints, where you prioritize experiments from your backlog every week, and choose what your team will be tackling. This ensures that your organization is continuously running experiments and iterating based on their success or failure. It is unlikely, with this method, that you or future hires will ever try the same thing twice.

The goal is to be constantly experimenting and ensure this culture is embedded in all aspects of your organization. Even at organizations that are at more advanced stages of growth, this activity can encourage creativity and critical thinking about the business, even when activities seem to be longer term and more predictable.

Experiment Template (Notion)

Growth experimentation is an exciting concept but difficult to execute upon if you don’t have the right framework or process. Start with a strong foundation via product marketing, and clearly outline your assumptions and hypotheses for an experiment. Next, choose and rank your channels for testing, and build an experiment using our sample brief. Once you launch the experiment, keep an eye on the milestones you set and ensure you’re running Growth Sprints to manage your experimentation backlog and iterate based on results.

Sounds like a lot to handle by yourself? Divisional can help—get in touch below and we’ll help you navigate your growth goals by connecting you to the people you need to achieve them.

Measuring KPIs is important for growing sales teams. Here are the top KPIs that every sales team should track in 2023.

Creating a content marketing strategy doesn't have to be complicated, yet many marketers make some common mistakes. Here are 7 steps to create a strategy.

Learn the difference between both types of outbound email, including examples and when to use them.

looking for a growth hire?