Design An A/B Test Analysis
Specifies the statistical analysis for an experiment, including SQL for metrics and significance interpretation.
Prompt
ROLE: You are an experimentation analyst who designs and analyzes A/B tests rigorously. CONTEXT: We ran an experiment testing [HYPOTHESIS]. Primary metric: [PRIMARY_METRIC]; guardrail metrics: [GUARDRAILS]. Assignment table and event tables: [SCHEMA]. Randomization unit: [UNIT]. Test ran [DATES]. Engine: [DATABASE_ENGINE]. TASK: 1. State the null and alternative hypotheses and the decision rule (alpha, one/two-sided, minimum detectable effect). 2. Write SQL to compute, per variant: sample size, the primary metric, variance, and guardrails at the randomization unit grain. 3. Choose the correct test (two-proportion z, t-test, or note when to use a ratio/delta method or CUPED) and justify it. 4. Check for sample-ratio mismatch and explain why it matters. 5. Interpret a plausible result: effect size, confidence interval, p-value, and the practical recommendation. OUTPUT FORMAT: Hypotheses & decision rule -> Metric ```sql``` -> Test choice & SRM check -> Interpretation template -> Pitfalls (peeking, novelty, multiple comparisons). CONSTRAINTS: Analyze at the randomization unit, not the event. Report confidence intervals, not just p-values. Flag peeking and multiple-comparison risks. State assumptions behind the chosen test.
How to use this prompt
- 1
Copy the prompt above and paste it into ChatGPT, Claude, or Gemini — or open it in the visual Studio to edit each part on a canvas and run it with your own key.
- 2
Replace any bracketed placeholders with your specifics. The more concrete your context and constraints, the sharper the result — see the 5-part prompt structure.
- 3
Run it, then refine. Ask the model to critique and improve its own answer with self-critique prompting.
Techniques in this prompt
Assigns the model an expert persona so it adopts the right vocabulary, depth, and standards for the task.
Learn this techniqueForces explicit intermediate reasoning instead of jumping to a conclusion, which improves accuracy on hard tasks.
Learn this techniquePins the response to a defined structure so it drops straight into your workflow.
Learn this techniqueRecommended models
Build on this prompt
Open it in the visual Studio to wire it into a full workflow with your own API key — or learn the craft behind prompts like this.
More in Data Analysis & SQL
Translate Business Questions Into SQL
Turns a plain-English stakeholder question into a correct, well-commented SQL query against a known schema.
Optimize A Slow SQL Query
Diagnoses why a query is slow and rewrites it with targeted, explained optimizations and an index plan.
Debug A SQL Query That Returns Wrong Results
Systematically finds the logic error producing incorrect numbers and delivers a corrected, verified query.
Explain An Unfamiliar SQL Query In Plain English
Reverse-engineers a complex inherited query into a clear narrative, business meaning, and risk list.