How to Calculate Incremental Revenue from an A/B Test: A Comprehensive Guide

Introduction

Calculating estimated incremental revenue from an A/B test is essential for understanding the financial impact of changes made to a website or app. By analyzing the data and ensuring statistical significance, marketers can make confident decisions backed by quantitative insights. In this guide, we’ll walk through how to determine which metrics to tie directly to A/B test results, differentiate causation from correlation, calculate statistical significance, and tell a story from the data.

We’ll use a hypothetical test as our example: comparing a multi-stage intake form to a single-stage form to see which one yields higher submission rates and, consequently, more revenue.

Example A/B Test Overview

Test Scenario:

A multi-stage intake form (Variant A) vs. A single-stage intake form (Variant B)

Test Duration: 59 days

Step-by-Step Statistical Significance Calculation

Statistical significance helps determine if the observed difference between the two variants is due to the test change or random chance. The following table provides a step-by-step demonstration of how to calculate this.

Table 1: Data Summary

Metric	Variant A (Multi-Stage)	Variant B (Single-Stage)
Total Sessions	10,000	10,200
Form Submissions	1,400	1,800
Submission Rate (%)	14%	17.65%

Step 1: Calculate the Conversion Rate for Each Variant

Conversion rate (CR) = (Number of Form Submissions / Total Sessions) * 100

Variant A (Multi-Stage Form):	CR = (1,400 / 10,000) * 100 = 14%
Variant B (Single-Stage Form)	CR = (1,800 / 10,200) * 100 ≈ 17.65%

Step 2: Calculate the Standard Error for Each Variant

Standard Error (SE) helps understand the variability of the conversion rate.

Calculations:
Standard Error for Variant A: ≈ 0.0035
Standard Error for Variant B: ≈ 0.0039

Step 3: Calculate the Z-Score

Z = (ConversionRate_B – ConversionRate_A) / √(SE_A² + SE_B²)

Substitute the values:
Z ≈ 7.84

Step 4: Find the p-value

A Z-score of 7.84 corresponds to a p-value close to 0, indicating a highly significant result well below the 0.05 threshold. To achieve statistical significance at a 95% confidence level, the Z-score must be at least 1.96. This threshold implies that there is only a 5% or lower probability that the observed difference is due to random chance.

A Z-score higher than 1.96 suggests that the results are significant and unlikely to be random. For higher confidence levels, such as 99%, the Z-score threshold rises to approximately 2.58, corresponding to a p-value of 0.01 or less.

Statistical Significance Calculation Summary

Metric	Variant A (Multi-Stage)	Variant B (Single-Stage)
Total Sessions	10,000	10,200
Form Submissions	1,400	1,800
Submission Rate (%)	14%	17.65%
Standard Error	0.0035	0.0039
Z-score	7.84
P-value	~0
Confidence Level	99%

Real-World Application: Incremental Revenue Calculation

To estimate incremental revenue, let’s assume the following:

12-month Average Revenue per Submission	$50
Incremental Conversions over 59 days	≈ 369 (+6.25423 submissions/day)

Estimated Incremental Revenue Accrued over 12 months:

$50 per form submission × (6.25423 submission increase/day × 365 days)

≈ $114,139.70 accrued over 12 months.

Conclusion

By calculating statistical significance and understanding the conversion metrics, we can confidently say that switching to a single-stage form led to a significant increase in form submissions. The estimated incremental revenue of $114,139.70 demonstrates the tangible value of optimizing the user journey based on A/B test results.

When conducting A/B tests, always watch for:

The difference between causation and correlation.
Proper sample size and test duration.
Clear data storytelling to align with business goals.

Using these steps, you can leverage data-driven insights to make impactful changes that boost your bottom line.