A/B testing whole funnels, not buttons
A per-page test measures a conditional probability that your own change moves. Here is the arithmetic for testing a funnel end to end instead.
Somebody is going to tell you to test your button colour. Ignore them, but ignore them for the right reason. The problem is not that button colour is a small effect. The problem is that on a multi-step funnel, a per-page test does not measure what the person running it thinks it measures.
Autonnel’s A/B testing is funnel-level. You designate one funnel as the host and bind other complete funnels as variants, each with a traffic percentage from 1 to 99, the host taking the remainder. There is no page-level split test: you cannot A/B test individual pages within a funnel. That constraint annoys people. I think it is correct, and this post is the argument for why.
What a per-page test actually measures
Take a funnel with a landing page, a checkout and an upsell. You test two versions of the checkout page and report “checkout conversion”. What you measured is:
P(purchase | this visitor reached the checkout page)
That conditional bar is doing enormous work. The set it conditions on is not a population you sampled, it is the output of everything upstream. If the landing page is byte-identical in both arms, that set is the same in expectation and the number is interpretable. Fine.
But nobody tests only the checkout. The interesting changes are upstream: a different promise on the landing page, price shown earlier or later, a qualifying question, a longer sales letter. Every one of those changes who arrives at the next step, and once the population changes, a step-level rate is no longer comparable across arms.
Here is the smallest example that shows the failure honestly. Two arms, 10,000 visitors each. The checkout page is exactly the same file in both arms. Nothing about it changed.
| Reach checkout | Checkout to purchase | Orders | End to end | |
|---|---|---|---|---|
| Host | 2,000 (20%) | 10.0% | 200 | 2.00% |
| Variant | 1,200 (12%) | 14.0% | 168 | 1.68% |
The variant states the price on the landing page. Fewer people click through, and the ones who do already accepted the price, so they convert better at checkout. The variant wins the checkout-step metric by 40% relative and loses the business by 16% relative.
Run as a “checkout page test”, you ship it. The dashboard said 14% versus 10%, and the dashboard was telling the truth about a number with no causal content, because the treatment moved the denominator.
It runs the other way too. A louder landing page pushing 32% through to a checkout converting at 7.5% produces 240 orders against the host’s 200: step metric down a quarter, business up a fifth.
This is the mechanism behind Simpson’s paradox: a rate computed inside a subgroup whose membership the treatment itself decided. The consequence people resist, stated flatly: step-level wins do not compose. If you test the landing and the checkout separately and each shows a lift, you cannot multiply them and you cannot add them. You have two conditional probabilities measured on populations that will not coexist.
The only metric that is immune is the one measured from a denominator the treatment cannot touch: visitors entering the funnel, divided into orders leaving it.
What that costs, stated with the assumptions attached
End-to-end measurement is correct and it is expensive, and anyone selling you the first half without the second half is selling you something.
The cost comes from the baseline rate. For a two-proportion test with equal arms, the standard sample size is:
n per arm = 2 * (z_alpha/2 + z_beta)^2 * p_bar * (1 - p_bar) / delta^2At a two-sided alpha of 0.05 and 80% power, (1.96 + 0.8416)^2 = 7.85, so the leading constant is 2 * 7.85 = 15.70. Everything below uses those assumptions. If you want 90% power or a one-sided test, the numbers move and you should recompute rather than quote mine.
| Baseline | Detect (relative) | Target | n per arm | Total visitors |
|---|---|---|---|---|
| 2.0% end to end | +20% | 2.4% | 21,110 | 42,220 |
| 2.0% end to end | +50% | 3.0% | 3,826 | 7,652 |
| 1.0% end to end | +20% | 1.2% | 42,695 | 85,390 |
| 30% at one step | +20% | 36% | 964 | 1,928 |
Working the first row so you can check me: p_bar = (0.020 + 0.024) / 2 = 0.022, so p_bar * (1 - p_bar) = 0.021516. delta = 0.004, so delta^2 = 0.000016. Then 15.70 * 0.021516 / 0.000016 = 21,110.
Now compare rows one and four. Same relative effect, same power, same alpha. The end-to-end test needs about twenty-two times the traffic, and the entire difference is that 2% is a smaller number than 30%. That is why per-page testing is popular: it is cheap because its denominator is pre-filtered, and it is pre-filtered by the thing you are trying to measure.
Put a clock on it. At 200 visitors a day into the funnel, row one is 42,220 / 200 = 211 days. Two seasons, one product cycle, and probably a pricing change in the middle. Row two, which asks only whether a genuinely different offer beats the old one by half, is 7,652 / 200 = 38 days.
So the honest advice for a store doing a couple of hundred visitors a day is not “test more”. It is: stop testing small changes, because you cannot resolve them, and running the test anyway does not make the answer appear. A test you were never powered to win produces a p-value that is a coin flip dressed as evidence. Shipping on it is worse than shipping on taste, because taste at least knows it is taste.
A lot of small-store A/B testing is theatre. Not fraud, not stupidity, just arithmetic that nobody did.
Two more taxes while we are counting. Running the host against three variants is three comparisons, and at 0.05 each the chance of at least one false positive is 1 - 0.95^3 = 14.3%. Correcting with Bonferroni pushes alpha to 0.0167, the z to about 2.394, and the leading constant from 7.85 to (2.394 + 0.8416)^2 = 10.47, which is 33% more traffic per arm. And peeking: if you check daily and stop the first time the result crosses 0.05, your false positive rate is far above 5%, because you gave yourself many chances to cross. Fix the horizon in advance, or use a sequential method built for it.
What the machinery does and does not give you
Autonnel assigns on first visit to the host funnel’s first landing page. New visitors get a weighted random draw against the configured percentages; returning tracking IDs are looked up and served the same funnel every time. Assignment records carry a 30-day TTL, the config itself a 12-hour TTL. Per-funnel numbers come from the analytics tab and from GET /api/v1.1/stats/funnels/:funnelId, which returns views, checkouts, purchases and upsell acceptance, counted by unique user.
Because each variant is a whole funnel, those per-funnel stats are already the end-to-end comparison. That is the point of the design: you are not assembling a funnel-level answer out of step-level fragments, the arm is the funnel.
Now the parts I would want to know before trusting a result.
Nothing here computes significance for you. The A/B documentation describes assignment, percentages and TTLs. It does not describe a significance test, a minimum sample threshold, or a stopping rule. Every calculation above is yours to do.
Randomization unit and analysis unit are documented separately. Assignment is keyed on tracking ID. Stats are counted by unique user. The docs do not state that these are the same identity, so do not assume it. Check it against your own data before you treat the stats count as your denominator; a mismatch there silently corrupts every rate in the test.
A variant with no landing page falls back to the host silently. This is the footgun that matters most, because the recommended way to build a variant is Duplicate, and a duplicate contains no landing pages. The default path to creating an arm produces exactly the state that quietly serves the host. Your split looks configured, traffic looks fine, and the experiment is comparing a funnel to itself.
Configuration takes up to 12 hours to fully take effect for unassigned visitors, so a mid-test split change gives you a transition window with two regimes in it. Do not change the split mid-test. If you must, discard that window.
Stickiness expires at 30 days. Reread the 211-day row. A test that long outlives its own assignment records, so a returning visitor can be re-randomized into the other arm and contribute to both. There is also no funnel versioning, so if someone edits a variant on day 90 nothing records that what you were measuring changed underneath you.
Removing a variant does not redistribute its traffic either; the remaining percentages stay put and no longer sum to 100 until you fix them by hand.
The rule I would actually follow
Test whole funnels, because that is the only unit whose denominator your change cannot move. Measure entrances to orders and nothing else. Before starting, compute the sample size from your real baseline and an effect you would actually act on, divide by your real daily traffic, and look at the number of days. If that is far past 30, do not run the test. Ship the bigger swing, or make a judgement call and label it one.
Small stores should test offers, prices and mechanisms, not layouts: only large effects are visible at their traffic, and only large effects were worth the calendar time anyway.
The mechanics are in funnel A/B test and the stats surface in the external endpoints reference. If you are still choosing a tool, the funnel builder comparison and pricing are the pages to read next, and I would treat page-level split testing as a feature to interrogate rather than a feature to want.