Showing two versions to different users at random and measuring which performs better, the honest way to settle design debates.
Why it exists
When you change something you rarely know whether the new version is actually better or just different, and shipping on a hunch can quietly hurt the metric you care about. A/B testing exists to show two versions to comparable groups and measure the difference, so a decision rests on evidence rather than the loudest opinion in the room.
How it actually works
Instead of arguing whether the green or blue button converts better, you show green to half your users and blue to the other half, at random, and measure. Whichever wins on the metric you chose wins the argument, with evidence.
Done properly it needs enough users to be statistically meaningful and a clearly chosen success metric. Feature flags are the machinery that splits traffic; events measure the result.
A senior PM walks you through it
A junior PM, stuck
We are testing a new "Order now" checkout button against the current "Add to cart" one. Two days in, the designer says the new button is clearly winning and wants to ship it now. I have the day-7 readout in front of me and I honestly cannot tell whether the numbers back that up or not, so I do not want to green-light a rollout on a feeling.
An A/B test is only worth running if you also know when it has actually decided, and "clearly winning after two days" is the single most common way teams fool themselves. Two things settle it: are the two groups really comparable, and is the gap between them bigger than random noise. Below is the exact query that built your readout and the day-7 table it returned, with the significance line spelled out, so you can answer the designer from the numbers.
The checkout-button test: readout query and day-7 result
The query behind the readout, run on day 7
>SELECT variant,
2 count(distinct user_id) AS users,
3 count(distinct user_id) FILTER (WHERE converted) AS conversions
4FROM experiment_assignments
5WHERE experiment = 'checkout_button_v2'
6GROUP BY variant;
Day 7 readout
7 variant | users | conversions | conversion
8 add_to_cart (current) | 8097 | 372 | 4.6%
9 order_now (new) | 8102 | 332 | 4.1%
10
11difference (new minus current): -0.5 points
1295% confidence interval on the difference: -1.1 to +0.1 points
13gap crosses zero: not statistically significant at day 7
Click a step to see the lines it points at.
Mistakes I've seen
Calling the test on day two. Early percentages swing on a few orders; a lead that looks clear at 300 users per group routinely vanishes by 8000. Peeking is fine, deciding on a peek is not.
Reading the point estimate and stopping. 4.1 versus 4.6 looks decisive until you see the confidence range crosses zero; without the significance line, a gap is just a number, not a result.
Ignoring whether the groups are comparable. If one variant got far more users or a different traffic source, the comparison is broken before you start; check the group sizes match first.
Moving the goalposts after seeing data. Picking the success metric, sample size, and end date up front is what makes the result trustworthy; choosing them once results are in is how teams talk themselves into shipping.
A/B testing something too small to ever reach significance. On low-traffic surfaces the test may never separate the two versions; knowing when to just decide instead of test is part of the judgment.
Tell the designer: "At day 7 the new button is at 4.1 versus 4.6, and the confidence range on that gap still crosses zero, so the test has not picked a winner. Two days was noise. Let it run to the planned sample before we ship." You read a readout as group sizes, a difference, and whether it clears the noise, which is the skill.
Where a PM meets this
A/B testing is how strong product teams make decisions with data instead of opinion; running and reading them well is a core skill.
It has limits: small user counts can't reach significance, and testing everything slows you down. Knowing when to test versus just decide is judgment.
Hear it in a meeting
"Let's A/B it, we'll split traffic and read conversion after a week."