Build a Conversion Test Log Your Team Learns From
Sets up a record of what you tested, what happened, and what you now believe — so wins get reused, losses aren't repeated, and the program compounds. Use it once, then keep it going after every test.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
You are an experimentation program lead. The difference between a team that runs tests and a team with an experimentation program is memory. Without a log, the same idea gets retested every eighteen months, losses get quietly forgotten, and nobody can say what the program actually taught the company.
**Our situation** — how many tests we run, who's involved, what tools: {{program_context}}
**What we've already tested** (paste whatever record exists, however messy): {{test_history}}
**Where documentation would live:** {{tooling}}
**Who needs to read it and what they need from it:** {{stakeholders}}
Produce:
1. **The entry template.** One per test, short enough that people will fill it in. It must capture: the hypothesis and the evidence behind it, what changed (with a screenshot reference), primary and guardrail metrics, sample size and dates, the result with its confidence and range, the validity checks that were run, the decision made, and — the field that matters most — **what we now believe about our customers**. A log of results is an archive; a log of beliefs is an asset.
2. **The classification scheme.** Tag each test by page or flow, funnel stage, the type of change (copy, layout, offer, friction, proof, price, speed), the audience segment, and the outcome (win / loss / inconclusive / invalid). This is what makes patterns findable later.
3. **Backfill what I gave you.** Convert my existing history into entries. Where information is missing, mark it explicitly rather than guessing — and note which gaps make an old result unusable.
4. **Find the patterns already there.** Across my history, look for: change types that consistently win or lose for us, pages that resist everything, segments that behave differently, results that contradict each other, and anything we tested twice. Say what my own data suggests about where to spend the next quarter — and be clear about how much weight a handful of tests can carry.
5. **Handle the awkward categories properly.**
- **Losses** get documented with equal care. A loss that stops a future team from spending a month on the same idea is worth as much as a win.
- **Inconclusive** results get recorded with the effect size the test could have detected, so we know what we ruled out versus what we simply couldn't see.
- **Invalid** tests get recorded as invalid with the reason, so nobody cites the number later.
- **Shipped-without-testing** changes get logged too, since they're the reason a later baseline moved.
6. **The review ritual.** A short monthly agenda: what shipped, what we learned, what it changes about the roadmap, and what we'd now stop doing. Plus a quarterly summary for leadership with the honest numbers — how many tests ran, how many won, and the estimated cumulative effect, with its caveats.
7. **Set the expectation.** Tell my stakeholders up front that most tests don't produce a clear win, that this is normal and not a sign of failure, and that the value comes from compounding small confirmed gains plus avoiding expensive mistakes. A program judged on win rate will start testing safe, trivial changes and stop learning anything.
Output the template as something I can paste directly into my tool, then the backfilled entries, then the pattern analysis.