Comprehensive End-to-End A/B Test Case Study for Content Optimization
This blog walks through how A/B testing can be used to design better digital experiences, using a real-world case study to illustrate the full experimentation lifecycle.
A/B testing is a powerful methodology widely used by companies to make data-driven decisions and achieve impactful results. E-commerce businesses, in particular, rely extensively on a test-and-learn culture to optimize user experiences and drive growth. In this blog, we will explore one such use case in detail.
We will start by categorizing A/B tests into different types. In this context, Control represents the existing experience, while Test/Variation refers to the new experience being evaluated. The majority of the tests we conduct can be classified into the following categories:
New Feature Launch on Digital Platforms (App/Web): These experiments are challenging because the control group often lacks a baseline, requiring decisions based on a comparable metric between the control and test groups (sometimes referred to as variations). Example: Introducing a new banner to drive signups on the website.
Content Optimization: After a feature is launched, experiments are run to optimize its content, such as the header message, images, or call-to-action (CTA) elements.
Targeted Experiences: These involve building and testing personalized experiences for specific user cohorts or segments. Example: Offering different pricing plans for students versus regular consumers.
Top-of-Funnel Traffic Optimization: Tests focus on different marketing channels and CRM campaigns to optimize the cost of acquisition (CAC).
Reducing User Friction: Experiments aim to simplify the user journey by reducing steps in the funnel, thereby improving conversion rates.
Store-Based Tests: Unlike user-based digital experiments, these require selecting comparable and time-stable counterfactual control stores with similar profiles to the test stores. These tests are also influenced by network or interaction effects. We’ll explore a detailed example of this category in future blogs.
In this blog’s detailed case study, we will explore an example that falls under content optimization. Please note that all data points referenced in this article have been anonymized to maintain confidentiality.
Phase 1 (Pre-Test) : Collect detailed information about the experiment and align with stakeholders on the problem statement, hypothesis, and proposed solution.
Experiment details
Problem Statement: Trial signups from a specific touchpoint have been declining over time, indicating a decrease in relevance. This highlights the need to optimize the content to drive an increase in signups.
Hypothesis: Changing the CTA from “Start” to “Claim” conveys a stronger message to users, creating a sense of urgency and encouraging immediate action.
Solution: We aim to A/B test the version with the “Claim” CTA while keeping all other variables constant to measure its impact on key metrics.
User Experience
Phase 2 (Pre-Test): Execute pre-test launch steps, including formulating the analytical hypothesis, defining measurement metrics, and conducting a power analysis to determine the required test duration.
Hypothesis Formulation:
For the given example:
Null Hypothesis (H₀): There is no difference in trial signup rates between the test and control groups.
Alternate Hypothesis (H₁): There is a difference in trial signup rates between the test and control groups.
Test Type: A two-tailed test is preferred since the direction of the impact (positive or negative) is unknown before the test.
Measurement Metrics
Measurement metrics are essential for evaluating the success of an A/B test by quantifying its impact across different dimensions. They help assess both direct and indirect effects of the intervention, ensuring comprehensive analysis. Metrics are categorized based on their role in the experiment:
Input Metrics: These are the direct outcomes of the intervention. In this use case, the input metric is the Click-Through Rate (CTR), which measures the impact of changing the call-to-action message from “Start” to “Claim” to create a sense of urgency and drive user engagement.
Output Metrics/Success Metrics: These reflect the changes driven by input actions and are critical for determining whether to proceed with the launch. For this use case, the Trial Signup Rate (calculated as Signups/Impressions) serves as the primary success metric to evaluate the effectiveness of the intervention.
Guardrail Metrics: These are metrics that are not expected to improve but are closely monitored to ensure they do not deteriorate beyond an acceptable threshold. In this use case, examples include key eCommerce metrics such as Gross Merchandise Value (GMV), Conversion Rate, and Bounce Rate.
Cannibalization Metrics: Metrics anticipated to decline as a result of the intervention but are accounted for within the success metric to evaluate the long-term net impact. In this use case, page-associated clicks serve as the cannibalization metric.
This flowchart illustrates the decision-making framework for determining next steps based on experiment metrics during an A/B test. The process begins by compiling all relevant metrics and evaluating the success metric. If the success metric shows a negative impact, a heterogeneity or strata analysis is conducted. If all strata display negative results, the feature is not launched. However, if some strata show positive outcomes, the recommendation is to learn from the findings and relaunch the feature with improvements. This ensures that valuable insights from the experiment are leveraged effectively.
If the success metric is positive, the guardrail metric is assessed next. While theoretically, a feature should not launch if the guardrail metric deteriorates beyond acceptable thresholds, a heterogeneity analysis is performed to account for the considerable time and effort invested in development. This additional due diligence ensures informed decision-making before rejection. If the guardrail metric is within acceptable limits, the cannibalization metric is reviewed. If cannibalization metrics indicate regression in related metrics, the long-term impact is evaluated. Should the long-term impact justify the trade-offs, the feature is launched. Otherwise, the focus shifts to learning and relaunching to enhance outcomes.
Power Analysis
Power analysis is a statistical method used in A/B testing to calculate the required sample size to detect a meaningful difference between variations confidently. The key parameters for our A/B test are:
Significance Level (Alpha): 5% (the probability of Type I error).
Power (1-Beta): 80% (the probability of detecting a true effect).
Minimum Detectable Effect (MDE): 5% (varies between 5%-10%, defined based on prior tests, industry benchmarks, or surveys. A conservative MDE is preferred for more reliable results, though it requires a larger sample size).
Baseline Conversion Rate (Output Metric): Derived from historical data, assumed to be 10% for this analysis.
Output of Power Analysis: The estimated sample size per variation is 60k. With a daily traffic volume of 10k and a 50% random split, the test duration is calculated at 12 days (~2 weeks). To ensure robust results, it’s recommended to run the test for at least two business cycles to mitigate novelty, primacy, or seasonal effects.
Before we move to Phase 3 feel to Subscribe:
Phase 3 (During Test): Execute test monitoring steps, including defining the experiment unit of measurement, launching at lower traffic, performing data validation, checking for contamination, addressing sample ratio mismatches, conducting an intermediate readout as a sanity check, and gradually ramping up to maximum traffic.
Experiment unit of measurement
This test utilized unique user ID as the treatment unit, as the experience was rendered for logged-in users with accurate user ID tracking. Network effects were not suspected since trial signup actions are independent, with no interference between users. Unlike store tests, where spillover effects are a concern, this setup was straightforward. In contrast, tests with session- or page-level randomization units require a different approach. For more insights, refer to Ron Kohavi’s excellent book “Online Controlled Experiments: A Practical Guide to A/B Testing”.
Experiment ramp plan
The test ramp-up plan varies depending on the criticality and potential impact of the experiment. High-stakes or regressive tests typically require a slower ramp-up to minimize risk, whereas content-based tests can adopt an aggressive ramp-up since the failure risk is limited. In this use case, the test began with 20% traffic allocated to both control and variation groups. An equivalent traffic split is usually preferred as it ensures balanced samples, enhancing statistical power and reducing selection bias for more reliable results.
Experiment set up validation
Ensure the correct experience is rendered for both test and control cookies.
Validate against data contamination — this test observed 2% contamination where users encountered both control and test experiences, primarily due to cache clearing or app uninstallation, which led to fresh cookie assignments. However, this level of contamination was within the acceptable threshold, so the test was not relaunched.
Additionally, the difference between test and control assigned samples remained within the acceptable range of ±1%, with no sample ratio mismatch (SRM). In cases where SRM is detected, further analysis is required to identify the root cause. If unresolved, the test must be relaunched, as SRM invalidates the results.
Experiment intermediate read out
Conducted around one week post-test launch.
The primary purpose is to validate experiment setup, confirm correct data flow, ensure proper instrumentation/tracking, and check for sample size mismatches.
For this use case, 60% of the estimated samples were collected within a week, indicating the test is on track to achieve the target of 60k samples per variation.
A major downside to intermediate readouts is stakeholders overanalyzing data prematurely, potentially leading to “peeking.” Results must be communicated carefully with clear disclaimers.
Plotting the p-value will reveal initial fluctuations, which stabilize as more samples are collected (below figure).
In this case, even though two days showed statistically significant lifts, it’s too early to make a decision. The test should only be concluded after the full two-week duration.
Phase 4 (Post-Test): Post-test launch analysis, including the final readout with analytics recommendations, trade-off discussions with stakeholders, and monitoring metrics post-100% launch.
Test Conclusion: The test ran for two weeks, collecting over 60k samples per variation. It demonstrated a significant positive lift of 8% in the primary metric, the trial signup rate, driven by a substantial 13% increase in CTR. Both practical and statistical significance were assessed using the decision matrix below, and the results were found to be significant on both fronts.
Guardrail Metrics Regression: Despite the success in the primary metric, guardrail metrics regressed significantly. Conversion rate (% of users who placed an order) dropped by 2.3%. Bounce rate (% of users who abandoned the site) increased by 3.6%.
Rule of Thumb: Generally, a regression in guardrail metrics suggests that the experience should not be launched. However, further investigation is crucial to understand the drivers of these negative impacts.
Heterogeneity Analysis:
A platform-level breakdown showed that the regression was primarily driven by iOS, while other platforms remained neutral.
Since iOS contributes the most to overall signups, it heavily influenced the overall negativity.
Root Cause Identification: Upon validating the iOS experience, it was discovered that a bug disabled the “close” button on the signup screen, causing users to get stuck. This led to an increase in bounce rates and negatively impacted conversion.
Analytics recommendation
Fix the bug on iOS and relaunch the test to validate the experience on that platform.
Launch the test experience at 100% on Web and Android, as these platforms showed positive results.
Monitor metrics post 100% launch
With Android and Web contributing to 60% of signups, the pro-rated lift in signups was observed after the experience was launched to 100%. This lift is reflected in daily signup dashboards, ensuring transparent tracking and monitoring.
In scenarios where the lift is marginally positive (<5%), the signup metrics may not consistently reflect the expected lift post-launch. This can result from alternate influencing factors, such as traffic fluctuations, seasonality, or the introduction of additional features, highlighting the importance of ongoing monitoring and context-based interpretation.
Phase 5 (Long-Term Impact):
To measure the sustained effectiveness of the feature, a 3% holdout group was maintained throughout the quarter to evaluate the cumulative impact of the feature launch. By comparing the trial signup rate and other metrics between the holdout and the rest of the population, alongside the influence of additional feature rollouts, the analysis provided insights into the long-term value and scalability of the intervention.
This blog outlines the comprehensive steps involved in conducting an A/B test across the pre-test, during-test, and post-test phases. The example above demonstrates testing a content change between control and variation. In upcoming blog posts, we will explore additional use cases, including new digital feature launches and store-based feature rollouts. Stay tuned for more insights!
This is gold! Thank you for explaining A/B testing step by step in detail. Thank you for resharing it. Hoping to see more blogs on fundamental topics.