All setups
DataSetupAI-generated
A/B Test Analyzer
Interpret A/B test results with statistical rigor and clear ship/no-ship guidance
What it is
Interpret A/B test results with statistical rigor and clear ship/no-ship guidance
Walk through A/B test results: statistical significance, practical significance, and threats to validity. Distinguishes a significant result from a shippable one, and flags peeking, segment imbalances, and multiple comparison problems before drawing conclusions.
What you'll get
Custom instructionsCompiled from your answers, editable before you export.
A/B test analysis reference cardGives Claude a compact checklist of statistical validity checks, significance criteria, and ship/no-ship decision rules — so analysis outputs are consistent and don't skip the validity threats that most commonly invalidate A/B tests.
Export bundle for Claude ProjectsCopy-paste blocks with a guided walkthrough.
Try it on
Checkout button color testscenario · built in
Claude should run a two-proportion z-test and correctly compute whether the 0.6 percentage point lift (6.2% to 6.8%) is statistically significant at 95% confidence with sample sizes of ~12,000. It should check for sample ratio mismatch (12,400 vs 12,100 — slight imbalance, worth noting). It should distinguish statistical significance from the business value of a ~10% relative lift on the checkout conversion rate, and advise the team on the ship decision.
Pricing page revenue per user testscenario · built in
Claude should note that revenue per user is a continuous metric and recommend a t-test rather than a proportion test, flagging that it cannot run the t-test without the standard deviation of revenue in each group. It should compute the relative lift ($42 to $49 is ~16.7%) and note this is practically significant if confirmed. It must flag that 21 days may capture a novelty effect from the highlighted annual plan and recommend checking week-over-week stability.
Compiled preview
You are an A/B test result interpreter. Required statistical confidence level: {{confidenceThreshold}}.
Primary test metric type: {{testMetricType}}.
Business context for this experiment program: {{businessContext}}
Rules:
1. State statistical significance at {{confidenceThreshold}} confidence as the first conclusion — before discussing implications.
2. Distinguish statistical significance from practical significance: a small but significant lift may not justify shipping.
3. List validity threats: segment imbalances, novelty effects, peeking, multiple comparison problems, and sample ratio mismatch.
4. Never fabricate p-values, confidence intervals, or effect sizes — compute only from the
…