A/B Test Design & Experiment Planning

Process

• Understand what the user wants to test (creative, audience, bidding, landing page)

• Build structured hypothesis using the framework below

• Calculate required sample size and estimated duration

• Recommend platform-specific test setup

• Define success criteria and measurement plan

Hypothesis Framework

Every test must start with a structured hypothesis:

IF we [change/action]

THEN [metric] will [increase/decrease] by [estimated %]

BECAUSE [reasoning based on data or insight]

Example:

IF we replace polished product shots with UGC creator videos

THEN Meta CTR will increase by 25-40%

BECAUSE Andromeda prioritizes diverse creative formats and UGC consistently outperforms polished in 2025-2026 benchmarks

Hypothesis Quality Checklist

• Single variable being tested (isolate the change)

• Specific metric defined (not "performance")

• Estimated effect size stated (needed for sample size calculation)

• Timeframe defined

• Success/failure criteria clear before launch

Statistical Significance Calculator

Required Sample Size (per variant):

n = (Z_alpha + Z_beta)^2 × 2 × p × (1-p) / MDE^2

Where:

- Z_alpha = 1.96 (for 95% confidence)

- Z_beta = 0.84 (for 80% power)

- p = baseline conversion rate

- MDE = minimum detectable effect (relative %)

Simplified lookup:

Per variant, 95% confidence, 80% power

Test Duration Estimator

Duration = Required Sample Size / Daily Traffic per Variant

Minimum duration: 7 days (capture weekly patterns)

Maximum recommended: 28 days (avoid seasonal drift)

Learning phase: Google 7-14 days, Meta 3-7 days, LinkedIn 7-14 days

Inputs needed:

- Daily impressions or clicks

- Number of variants (2 = A/B, 3+ = multivariate)

- Baseline conversion rate

- Minimum detectable effect desired

Duration Quick Estimates

*Minimum 7 days recommended regardless of sample sufficiency

Platform-Specific Test Setup

Meta Experiments

• Use Ads Manager > Experiments tab (not manual ad set duplication)

• Automatic audience splitting ensures no overlap

• Supported test types: A/B (creative, audience, placement), Holdout, Brand Survey

• Meta's Incremental Attribution (April 2025) provides AI-powered holdout testing for measuring real causal impact

• Budget: split evenly across variants; minimum $100/day per variant recommended

• Duration: 7-14 days typical; Meta auto-determines winner at 95% confidence

Google Experiments

• Campaign Experiments (custom experiments) or Ad Variations

• Create experiment from existing campaign > select experiment type

• Traffic split: 50/50 recommended for fastest results

• Supported: bidding strategy, ad copy, landing page, audience

• Metrics: choose primary metric (conversions, CPA, ROAS) before launch

• Duration: 14-30 days recommended; minimum 2 weeks for bidding tests

LinkedIn A/B Testing

• Built into Campaign Manager for Sponsored Content

• Duplicate ad set with single variable change

• Target: same audience segment with automatic rotation

• Minimum budget: $50/day per variant

• Key metrics: CTR (>0.44% benchmark), CPL, Lead Form CVR (13% benchmark)

• Duration: 14-21 days (LinkedIn's smaller daily volumes require longer tests)

TikTok Split Testing

• Available in TikTok Ads Manager > Create A/B Test

• Test types: targeting, bidding, creative

• Auto-splits audience to avoid contamination

• Minimum 7 days, recommended 14 days

• Budget: minimum $20/day per ad group

• Creative tests: isolate hook (first 2-3 seconds) as the primary variable

• TikTok's enhanced split testing supports modular test variables (targeting, creative, budget, placement) via Smart+ since 2025

What to Test (Priority Order)

High Impact (test first)

• Creative concept (different messaging angles, not just color changes)

• Hook/first 3 seconds (video opening on Meta, TikTok, YouTube)

• Offer structure (pricing, discount type, free trial length)

• Landing page (headline, CTA, form length)

• Bidding strategy (tCPA vs tROAS vs Maximize Conversions)

Medium Impact

• Audience targeting (interest vs lookalike vs broad)

• Ad format (static vs video vs carousel)

• CTA button (Learn More vs Sign Up vs Shop Now)

• Campaign structure (CBO vs ABO, consolidated vs segmented)

Low Impact (test last)

• Ad scheduling (time of day, day of week)

• Device targeting (mobile vs desktop)

• Minor copy variations (word substitutions without concept change)

Common Testing Mistakes to Avoid

• Testing too many variables at once (no clear winner attribution)

• Ending tests too early (before statistical significance)

• Testing during atypical periods (holidays, launches, incidents)

• Comparing unequal time periods

• Not documenting learnings (build institutional knowledge)

• Testing small changes when big changes are needed (optimize vs innovate)

• Ignoring learning phase on automated platforms

Output Format

## A/B Test Plan

### Hypothesis

IF [change]

THEN [metric] will [direction] by [amount]

BECAUSE [reasoning]

### Test Design

| Parameter | Value |

|-----------|-------|

| Platform | [platform] |

| Test Type | [A/B / Multivariate] |

| Variable | [what's being changed] |

| Control | [current state] |

| Variant | [proposed change] |

| Primary Metric | [KPI] |

| Traffic Split | [50/50 / other] |

### Sample Size & Duration

| Metric | Value |

|--------|-------|

| Baseline CVR | [X%] |

| MDE | [X%] |

| Required Sample | [N per variant] |

| Daily Traffic | [N clicks/day] |

| Est. Duration | [X days] |

| Min Duration | 7 days |

### Success Criteria

- Winner declared at 95% confidence

- [Primary metric] improvement of [X%]+ sustained over [Y] days

- No negative impact on [secondary metric]

### Setup Instructions

[Platform-specific step-by-step]