Experiment Design

Objective: Deliver a reusable AI prompt structure that produces a complete experiment design, including a clear hypothesis, primary and secondary metrics, sample size calculation logic, segmentation strategy, guardrail metrics, and mitigation steps for potential confounding variables.

The base prompt was formed through these steps:

Identify the core need for Chain-of-Thought reasoning so the AI builds the plan sequentially rather than jumping to conclusions.
Separate fixed instructional language from changeable context so the same template works across products, channels, and teams.
Isolate reusable guidance on statistical rigor, metric hierarchy, and risk controls as snippets that can be updated independently.
Expose only the variables that differ per experiment: the hypothesis statement, business context, and desired tone of the output.
Enforce a consistent output order that mirrors real experiment documentation standards used in product and growth teams.

Note

Keep the hypothesis specific and falsifiable. Vague statements reduce the quality of every downstream calculation the model produces.

Element	Type	Example Value
{{snippet:experiment_framework}}	Snippet	Follow a structured Chain-of-Thought process. First restate the hypothesis. Then define primary and secondary metrics with clear success criteria. Next calculate required sample size using standard power analysis assumptions. After that outline segmentation strategy. List guardrail metrics that must not degrade. Finally identify potential confounding variables and concrete mitigation steps.
{{snippet:statistical_guidelines}}	Snippet	Use 80% power, 5% significance level, and two-sided tests unless otherwise specified. State all assumptions explicitly.
{{snippet:output_format}}	Snippet	Present the final plan in clearly labeled sections with bullet points for readability.
{{hypothesis}}	Variable	Adding a one-click reorder button on the order confirmation page will increase 30-day repeat purchase rate by at least 8%.
{{business_context}}	Variable	E-commerce fashion retailer with average order value of $85 and current 30-day repeat rate of 18%.
{{tone}}	Variable	precise and data-driven

Prompt Template

You are an expert experiment designer. {{snippet:experiment_framework}}

{{snippet:statistical_guidelines}}

Business context: {{business_context}}

Hypothesis to evaluate: {{hypothesis}}

{{snippet:output_format}}

Respond in a {{tone}} manner.

Prompt Example

You are an expert experiment designer. Follow a structured Chain-of-Thought process. First restate the hypothesis. Then define primary and secondary metrics with clear success criteria. Next calculate required sample size using standard power analysis assumptions. After that outline segmentation strategy. List guardrail metrics that must not degrade. Finally identify potential confounding variables and concrete mitigation steps.

Use 80% power, 5% significance level, and two-sided tests unless otherwise specified. State all assumptions explicitly.

Business context: E-commerce fashion retailer with average order value of $85 and current 30-day repeat rate of 18%.

Hypothesis to evaluate: Adding a one-click reorder button on the order confirmation page will increase 30-day repeat purchase rate by at least 8%.

Present the final plan in clearly labeled sections with bullet points for readability.

Respond in a precise and data-driven manner.

Tip

After generating the plan, ask the model to critique its own sample-size assumptions and list any missing data that would improve accuracy.

Extended example values:

{{snippet:experiment_framework}}
- Follow a structured Chain-of-Thought process. First restate the hypothesis. Then define primary and secondary metrics with clear success criteria. Next calculate required sample size using standard power analysis assumptions. After that outline segmentation strategy. List guardrail metrics that must not degrade. Finally identify potential confounding variables and concrete mitigation steps.
- Apply sequential reasoning: restate hypothesis, select metrics, compute sample size, design segments, set guardrails, and address confounders with mitigation actions.
{{snippet:statistical_guidelines}}
- Use 80% power, 5% significance level, and two-sided tests unless otherwise specified. State all assumptions explicitly.
- Default to 80% statistical power and alpha of 0.05. Document every assumption about baseline rates and minimum detectable effect.
{{snippet:output_format}}
- Present the final plan in clearly labeled sections with bullet points for readability.
- Structure the response with numbered headings matching the required plan components and use concise bullets under each.
{{hypothesis}}
- Adding a one-click reorder button on the order confirmation page will increase 30-day repeat purchase rate by at least 8%.
- Introducing personalized product recommendations on the homepage will lift average session conversion rate by 5% or more.
- Changing the primary call-to-action color from blue to orange will raise click-through rate on the pricing page by at least 12%.
{{business_context}}
- E-commerce fashion retailer with average order value of $85 and current 30-day repeat rate of 18%.
- SaaS project-management tool with free-to-paid conversion rate of 4.2% and average monthly churn of 3.1%.
- Mobile banking app serving 2.4 million active users with current feature adoption of the new budgeting tool at 11%.
{{tone}}
- precise and data-driven
- concise and actionable
- formal and statistically rigorous