content-experimentation-best-practices

Name: content-experimentation-best-practices
Author: sanity-io

Structured guidance for designing, executing, and analyzing content experiments to improve conversion and engagement. Covers hypothesis frameworks, metric selection, sample size calculation, and statistical significance testing across A/B and multivariate experiments Includes detailed resources on p-values, confidence intervals, power analysis, and Bayesian methods for interpreting results Provides CMS integration patterns for managing variants at the field level and connecting external experimentation platforms Documents 17 common pitfalls in experiment design, statistical analysis, execution, and result interpretation to avoid flawed conclusions

INSTALLATION

npx skills add https://github.com/sanity-io/agent-toolkit --skill content-experimentation-best-practices

Run in your project or agent environment. Adjust flags if your CLI version differs.

SKILL.md

Content Experimentation Best Practices

Principles and patterns for running effective content experiments to improve conversion rates, engagement, and user experience.

When to Apply

Reference these guidelines when:

Setting up A/B or multivariate testing infrastructure

Designing experiments for content changes

Analyzing and interpreting test results

Building CMS integrations for experimentation

Deciding what to test and how

Core Concepts

A/B Testing

Comparing two variants (A vs B) to determine which performs better.

Multivariate Testing

Testing multiple variables simultaneously to find optimal combinations.

Statistical Significance

The confidence level that results aren't due to random chance.

Experimentation Culture

Making decisions based on data rather than opinions (HiPPO avoidance).

References

Start with the reference that matches the current problem, such as design, statistics, CMS integration, or pitfalls. See references/ for detailed guidance:

references/experiment-design.md — Hypothesis framework, metrics, sample size, and what to test

references/statistical-foundations.md — p-values, confidence intervals, power analysis, Bayesian methods

references/cms-integration.md — CMS-managed variants, field-level variants, external platforms

references/common-pitfalls.md — 17 common mistakes across statistics, design, execution, and interpretation