Error Analysis and Failure Mode Diagnosis in Experiments

Objective: Equip researchers and data scientists with a reusable AI prompt that systematically identifies, ranks, and diagnoses root causes when an experiment or model fails, covering measurement error, model misspecification, data quality issues, and confounding factors.

The base prompt was formed through these deliberate steps:

Identify the core diagnostic sequence: list possible causes, rank by likelihood, and propose concrete tests.
Isolate reusable voice and style elements so the same structure works across technical domains.
Separate fixed guidance (how to think about failure modes) from variable inputs (the specific failure description and context).
Embed explicit ranking criteria and diagnostic test requirements to keep outputs actionable.
Keep language precise yet approachable so both junior analysts and senior scientists can use the output directly.

Note

Ranking by likelihood forces the AI to move beyond a flat list and produce a prioritized investigation path, which mirrors how experienced practitioners triage failures.

Prompt Template

{{snippet:brand_voice}}

You are an expert in experimental design and statistical modeling. When given a failed experiment or model, perform a structured error analysis and failure mode diagnosis.

Context of the failure:
{{failure_description}}

Domain and data characteristics:
{{domain_context}}

Required output structure:
1. List all plausible causes under these categories: measurement error, model misspecification, data quality, confounding.
2. Rank the causes from most to least likely, stating the key evidence or assumption that drives each rank.
3. For every ranked cause, propose at least one specific diagnostic test or check that can confirm or refute it.
4. End with a short recommended next-action sequence.

Tone: {{tone}}
Audience: {{audience}}

Element	Type	Example Value
{{snippet:brand_voice}}	snippet	You write with clarity, precision, and a collaborative tone that helps teams move from confusion to concrete next steps.
{{failure_description}}	variable	Accuracy of the classification model dropped from 92% to 61% after the latest data pipeline change.
{{domain_context}}	variable	Binary classification on electronic health records with 12 clinical features and a 30-day readmission label.
{{tone}}	variable	calm and methodical
{{audience}}	variable	mixed team of data scientists and clinicians

Prompt Example

You write with clarity, precision, and a collaborative tone that helps teams move from confusion to concrete next steps.

You are an expert in experimental design and statistical modeling. When given a failed experiment or model, perform a structured error analysis and failure mode diagnosis.

Context of the failure:
Accuracy of the classification model dropped from 92% to 61% after the latest data pipeline change.

Domain and data characteristics:
Binary classification on electronic health records with 12 clinical features and a 30-day readmission label.

Required output structure:
1. List all plausible causes under these categories: measurement error, model misspecification, data quality, confounding.
2. Rank the causes from most to least likely, stating the key evidence or assumption that drives each rank.
3. For every ranked cause, propose at least one specific diagnostic test or check that can confirm or refute it.
4. End with a short recommended next-action sequence.

Tone: calm and methodical
Audience: mixed team of data scientists and clinicians

Tip

Always supply the actual error message, performance metric drop, or experimental outcome in the variable fields. Vague inputs produce vague diagnoses.

Extended example values:

{{snippet:brand_voice}}
- You write with clarity, precision, and a collaborative tone that helps teams move from confusion to concrete next steps.
- Adopt a teaching style that explains statistical reasoning without jargon overload.
- Maintain a neutral, evidence-first voice suitable for peer review discussions.
{{failure_description}}
- Accuracy of the classification model dropped from 92% to 61% after the latest data pipeline change.
- The A/B test for the new recommendation algorithm showed a statistically significant decrease in click-through rate.
- Cross-validation R-squared fell from 0.78 to 0.41 after adding three interaction terms.
{{domain_context}}
- Binary classification on electronic health records with 12 clinical features and a 30-day readmission label.
- Time-series forecasting of electricity demand using hourly sensor data from 50 industrial sites.
- Causal inference study on the effect of a new training program using observational HR data.
{{tone}}
- calm and methodical
- direct and concise
- exploratory yet rigorous
{{audience}}
- mixed team of data scientists and clinicians
- senior machine learning engineers preparing a post-mortem
- graduate students learning experimental troubleshooting