Computation is free now, so the job is everything around it — the definition, the sample, the confound, the check. An arithmetically correct number can still be completely wrong.
Computation is free. Everything around it is the job.
Anyone can produce a number in seconds now. Write the query, run the notebook, ask a model. What nobody can do by looking at the output is tell whether the definition was right, whether the sample was representative, whether a confound is driving the result, or whether the effect is larger than the noise.
So the analyst's job has moved. It is no longer producing the number. It is the specification before and the verification after — and the middle part, the part that used to be the skill, is the part that got automated.
An arithmetically correct number can still be completely wrong
The clearest illustration is Simpson's paradox: an aggregate can point in the exact opposite direction from every single underlying segment. Treatment A beats treatment B overall, and B beats A in every subgroup. No error, no warning, no anomaly in the data. The query ran fine.
This is not an exotic edge case — it appears whenever group sizes shift between the periods or populations you are comparing, which in business data is most of the time. A conversion rate that improved
after a channel-mix change, an average order value that rose because the cheap segment stopped buying, a support satisfaction score that fell because you started surveying a harder cohort.
The practical consequence is that the number went up
is never a finding on its own, and mix-shift checks belong in the diagnostic work rather than in a footnote. Work Out Why a Number Moved is built around decomposition — separating a change in rate from a change in mix before proposing any explanation. Sanity-Check an Analysis Before You Share It is the final gate, and it is the prompt most worth running on work you are confident about.
Preparation is the work
The folkloric claim is that data scientists spend 80% of their time cleaning data. The honest version is a range: studies put it somewhere between about 25% and 80% depending on what you count as preparation and who you ask. CrowdFlower's survey work found around 60%; a 2020 follow-up put it nearer 45%.
We use the contested range deliberately rather than the tidy 80%, because the point survives either way — a large and irreducible share of analysis is getting the data into a state where the computation means something. Pretending otherwise sets up every analysis to be late and every estimate to be wrong.
Clean and Validate a Messy Dataset is systematic about it: types, ranges, duplicates, missingness patterns, and — the one people skip — whether missingness is random or structural, because structurally missing data is a finding rather than a nuisance.
Reconcile Two Data Sources That Disagree handles the situation that eats whole days. Two systems report different revenue and both are right
under their own definitions. The prompt works down the definitional stack — timing, currency, cancellations, test accounts, attribution window — instead of hunting for a bug that does not exist.
Write a Data Dictionary for a Table or Dataset and Design an Event Tracking Plan Before You Build the Feature are the upstream fixes. Most reconciliation work is a bill for tracking that was never specified. Defining the events before the feature ships costs an hour; recovering the definition afterwards costs a quarter of ambiguous data you cannot retroactively fix.
Ask the question before you pull the data
Write an Analysis Plan Before You Pull the Data exists to prevent the most common way analysis goes wrong, which is not a mistake in the maths. It is running many cuts, finding one that looks interesting, and reporting that one — which produces a compelling and unreproducible story every single time, without anyone intending to mislead.
Writing down the question, the population, the metric definition, and what result would change the decision, before you look, is what separates analysis from search.
Estimate a Number You Have No Data For covers the opposite case: no data at all. Fermi decomposition, explicit assumptions, and a stated range are far more useful to a decision than a precise-looking figure with a hidden guess inside it.
Is the difference real
Check Whether a Difference Is Real or Just Noise is scoped deliberately to observational comparisons — segments, periods, cohorts — rather than controlled experiments, which the CRO pack owns. That distinction matters, because the standard statistical machinery assumes randomisation you do not have when comparing last month to this month, and the failure mode is confidently declaring a difference that is a seasonal pattern with a p-value attached.
Run a Cohort and Retention Analysis is the single most clarifying reframing available to most businesses, because a blended retention number mixes cohorts acquired under completely different conditions. Segment Customers From Behavior, Not Intuition does the same for segmentation — behavioural segments predict; demographic segments describe.
Communicating without lying
Choose a Chart That Tells the Truth covers the choices that mislead without any false data: truncated axes, dual axes implying a relationship, aggregating away the distribution, and using a chart type whose visual encoding does not match the comparison being made.
Design a Dashboard People Will Actually Use starts from the decisions the dashboard should support. A dashboard that shows everything available gets checked twice and abandoned.
Write a Survey That Produces Usable Data covers instrument design — leading questions, double-barrelled items, scales that cannot be compared, and sampling that guarantees the answer before anyone responds.
Two prompts here come from adjacent packs and belong: Analyze a CSV or Dataset as the general-purpose entry point, and Report Your Impact Without Overclaiming, which is the nonprofit sector's version of the same discipline — sorting claims into output, outcome and impact, and refusing to promote any of them above their evidence.
Where this stops
These prompts work only from data you supply, and they are written not to invent figures, sources, or benchmarks. They will identify what to verify rather than assert that verification happened. Nothing here is a substitute for a statistician on a question where the answer carries real consequence — and any analysis touching personal data should clear your privacy obligations before it is run, not after.
Sources
- On the preparation share of analytics work: CrowdFlower's data science surveys (~60%) and subsequent replications putting it closer to 45% — a contested range rather than the commonly quoted flat 80%
- Simpson's paradox as a standard result in statistics; see any treatment of confounding and aggregation in observational data