Growth Experiment

What is a growth experiment?

Growth experiment (German: Wachstumsexperiment) refers to a time-limited trial with a limited commitment of resources, with which a company tests an assumption about its growth. The subject can be an acquisition channel (the route through which new customers are won), a target group, a price or a step in the purchase process. It differs from mere trial and error through three definitions made before the start: the hypothesis (the assumption to be tested), the success criterion and the end.

Growth experiments belong to the toolkit of marketing, sales and product development, especially in companies with product-led growth (PLG), in which customers try the product themselves and often buy without a sales conversation. They apply at all stages of the sales funnel, that is, the sales process from winning prospects through closing the contract to renewal. For investors, documented experiments are evidence that a growth channel has not only been claimed but measured, and that the team tests assumptions systematically.

Structure of an experiment

• Hypothesis: the assumption, its reasoning and the expected effect, for example that medium-sized industrial customers respond better to an approach through the maintenance management than through the managing director.
• Intervention: what exactly is changed and to what extent, preferably only one variable at a time.
• Primary metric and success criterion: the metric that decides, and the value from which the trial counts as passed.
• Counter-metric: a metric that must not deteriorate, for example the closing rate or the churn rate.
• Duration and commitment: time, budget and working hours; four to six weeks is common.
• Decision rule: scale up, adjust or end, depending on the result.

Test design and informative value

The most reliable design is the A/B test: the target group is randomly split into a test group with the change and a control group without it, and both run at the same time. This way, season, market conditions and parallel campaigns affect both groups equally. Comparisons of periods before and after a change without a control group are more susceptible to such influences.

The group size must match the expected effect: the smaller the difference, the more cases are needed for it to stand out from chance. In business with corporate customers and a few hundred approaches, therefore, only clear effects can be statistically secured. In addition there is the risk of early termination: anyone who watches the result continuously and stops at the first favourable interim position easily mistakes random fluctuation for an effect. Duration and group size are therefore fixed beforehand.

Legal limits

Experiments with users and prospects take place within the framework of data protection and competition law. Test tools that store or read information on users' end devices, for example via cookies, in principle require consent. This follows from Section 25 of the German Telecommunications Digital Services Data Protection Act (Telekommunikation-Digitale-Dienste-Datenschutz-Gesetz, TDDDG). The only exception is what is strictly necessary for the service requested by the user. Where personal data are processed, the General Data Protection Regulation (GDPR, German: DSGVO) also applies.

Advertising by email requires, under Section 7 of the German Act against Unfair Competition (Gesetz gegen den unlauteren Wettbewerb, UWG), prior express consent, also towards companies; narrowly limited exceptions apply for existing customers. Unsolicited emails to new contacts are therefore generally ruled out as a test channel. For advertising calls to companies, by contrast, presumed consent suffices, that is, a recognisable objective interest of the person called.

Significance for a funding round and due diligence

If a company justifies a funding round with a new growth channel, investors demand evidence. Documented experiments supply the figures that matter: conversion rates per stage, customer acquisition cost (CAC) per channel and key figures derived from them. The CAC payback states after how many months the gross profit of a customer has covered its acquisition costs. The LTV/CAC ratio relates the value of a customer over the entire customer relationship to these costs. In commercial due diligence, the review of market, customers and sales, the values are reconciled with actual data from the sales system and accounting. Cohort analyses show how customers from the same starting month behave over time.

Particular attention goes to transferability. A channel that shows good values with a small budget often becomes more expensive with greater spend, because the most receptive target groups are reached first. A financial model that extrapolates test results unchanged to a multiple of the budget is therefore regarded as too optimistic.

Worked example

A software company tests whether the offer of a free initial analysis generates more appointments with prospects. 800 companies that have requested information material via the website are randomly split into two equal groups and approached by sales. The success criterion is an appointment rate at least 50% higher in the test group. The counter-metric is closings per approach, which must not fall. The experiment runs for six weeks.

• Control group: 400 approaches without the offer, 24 appointments, appointment rate 24 / 400 = 6.0%
• Test group: 400 approaches with the offer, 44 appointments, appointment rate 44 / 400 = 11.0%
• Relative increase: (11.0 minus 6.0) / 6.0 = 83.3%; the success criterion is met
• Closings: control group 6 of 24 appointments (25.0%), test group 7 of 44 appointments (15.9%)
• Counter-metric: 6 / 400 = 1.50% compared with 7 / 400 = 1.75% closings per approach; it did not fall, but the difference amounts to only one customer
• Additional effort: 44 initial analyses × 3 hours = 132 hours, at internal costs of EUR 80 per hour EUR 10,560
• Additional annual contract value: 1 customer × EUR 12,000 = EUR 12,000

The difference in appointments is statistically significant: without a real effect, such a large gap would arise by chance in only about 1 to 2% of cases, that is, less often than the customary 5%. The difference of one closing, by contrast, cannot be distinguished from chance. Of the 20 additional appointments, only one led to a contract; the closing rate per appointment fell from 25.0 to 15.9%. An additional annual contract value of EUR 12,000 is set against an effort of EUR 10,560. The appropriate decision is therefore “adjust”: limit the offer to the segment with the highest closing rate and test again.

Distinction from related terms

• A/B test: a test design with test and control group. A growth experiment can be set up as an A/B test but additionally includes hypothesis, decision rule and documentation.
• MVP: tests whether a product meets a need. Growth experiments usually come later and test how an existing product grows economically.
• Market analysis: surveys market size, competition and customer statements. An experiment measures what customers actually do.
• Go-to-market strategy: fixes target groups, channels and prices; experiments supply the evidence for these decisions.

Common mistakes

• No success criterion before the start. If the yardstick is chosen only after the trial, something encouraging can be found in almost every result.
• Several changes at once. If approach, target group and price are changed together, the result cannot be attributed to any cause.
• No counter-metric. An improved intermediate stage can worsen the closing without anyone noticing.
• No documentation. Without a log, nobody remembers after a year what has already been tested, and investors lack the evidence.
• Extrapolating without testing. Results from a small test budget are transferred to ten times the budget without a further trial.

Frequently asked questions

How many experiments sensibly run at the same time?

Usually two to three, and only if they do not influence each other. Several experiments at the same stage of the purchase process distort one another.

How large must a test group be?

Large enough for the expected difference to stand out from chance. A rise from 6 to 11% can already stand out with 400 cases each, as the worked example shows. To detect a rise from 6 to 7% with a probability of 80%, around 9,500 cases per group would be needed.

What happens to failed experiments?

They are documented and ended. A cleanly failed experiment rules out an option and spares later discussions about the same approach.

Do A/B tests on the website need consent?

As a rule yes, if the tool used stores or reads information on the end device. The only exception is what is strictly necessary for the service expressly requested by the user.

The Roemer Capital view

If a funding round is justified with a new growth channel, Roemer Capital takes the results of documented experiments into the financial model as assumptions: conversion rates, customer acquisition costs and ramp-up times per channel. In the cautious case, Roemer Capital calculates only with channels that are evidenced in experiments or in the ongoing business; untested channels appear at most in the optimistic case.

As the yardstick for a viable channel, Roemer Capital applies an LTV to CAC ratio of at least 3 to 1 and a CAC payback of under twelve months. Two to three documented experiments that support these values are more reliable in an investor meeting than a market study, because they show that the team tests assumptions before it commits capital.