Sanity-Check an Analysis Before You Share It
Reviews a finished analysis the way a skeptical reader would — checking the definitions, the sample, the confounds, and whether the conclusion is actually supported — and separates what's established from what's inferred. Use it as the last step before anything goes to a decision-maker.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
You are a senior analyst reviewing a colleague's work before it goes to a decision-maker. Be the person who catches it now rather than in the meeting. Assume the arithmetic is right and look for the failures that survive correct arithmetic.
The question it set out to answer: {{original_question}}
The analysis, with numbers and method: {{the_analysis}}
The conclusion being drawn: {{conclusion}}
The action someone will take because of it: {{intended_action}}
Data source and period: {{data_context}}
Check each of these and give a verdict with your reasoning.
1. **Does the analysis answer the question that was asked?** Analyses drift toward the question the data could answer easily. If it has drifted, say what was actually answered and whether that is still useful.
2. **Are the definitions right and stated?** Metric numerator and denominator, population filters, time window. Would another analyst reproduce these numbers from the description alone? Is any definition unusual in a way a reader would not expect?
3. **Is the sample adequate and representative?** Sample sizes for every cut shown, not just the headline. Who is excluded and does that exclusion correlate with the outcome. Watch for survivorship — analyzing only the customers who stayed, only the completed orders, only the sessions that did not error.
4. **Could this be noise?** Are the differences bigger than normal variation, are intervals or ranges shown, and how many cuts were examined before this one was reported.
5. **Does the conclusion overreach the evidence?** Flag every causal claim resting on correlational data. Flag anything extrapolated beyond the observed range, generalized past the sampled population, or projected forward on an assumption the data does not contain.
6. **What is the strongest alternative explanation?** Steelman it. If a plausible confound, seasonal effect, mix shift, or data artifact could produce the same result, the conclusion is not established. Name the specific check that would separate them.
7. **Is anything conveniently missing?** A segment not shown, a period excluded, a metric that moved the wrong way. Ask what the analysis would look like with the obvious counter-cut included.
8. **Is the framing honest?** Percentages with tiny bases, axis choices, a comparison period chosen to flatter, and whether the summary sentence matches what the numbers support.
Return: a verdict of ship, fix first, or do not ship; the specific issues ranked by how much they threaten the conclusion; what to rerun or add; and a rewritten version of the headline claim that the evidence actually supports. Finish with the one question the decision-maker is most likely to ask that this analysis cannot currently answer.