A/B Testing
Interpreting Results
Read SortLab A/B test revenue, units, confidence, days active, and winner state before applying a strategy.
When a test is running or complete, SortLab shows both variants side by side so you can decide whether the challenger deserves to become the collection's active strategy.
Result Fields
| Field | Meaning |
|---|---|
| Control (A) | The strategy that was active before the test started. |
| Challenger (B) | The strategy configured during test creation. |
| Revenue | Product revenue attributed to each variant's active windows. |
| Units sold | Product units sold while each variant was active. |
| Orders | Orders attributed to the tested collection and variant window. |
| Days active | How long each variant was active. |
| Revenue per day | Revenue normalized by active exposure time. |
| Units per day | Unit velocity normalized by active exposure time. |
| Confidence | Whether the observed difference is likely to be real. |
SortLab identifies the winner by revenue per day when the confidence threshold is high enough.
Reading Confidence
Confidence helps separate a real signal from normal traffic noise.
| Confidence state | What to do |
|---|---|
| Insufficient | Keep the test running. There is not enough data yet. |
| Moderate | Treat the result as directional, not final. |
| Good | There is a strong signal, but review business context. |
| Significant | The evidence is strong enough to act on in most cases. |
Above 95% confidence is generally considered statistically significant. Low-traffic collections may need longer tests to reach that level.
Decide What To Apply
- Check the winner and confidence. Do not rely on raw revenue alone.
- Compare revenue per day and units per day. These normalize for exposure time.
- Review the daily comparison. Look for consistent performance rather than one unusual spike.
- Consider business context. A strategy with slightly less revenue but better margin, inventory relief, or product launch exposure may still be useful.
- Apply the winner. If auto-apply winner was enabled, verify the final strategy on the collection. If it was disabled, apply the winning configuration manually.
When To Ignore or Re-run a Result
Re-run or disregard a test if:
- A major sale, outage, campaign, or inventory event affected only part of the test.
- Product membership changed substantially during the test.
- A variant had too little active exposure.
- Confidence never moved beyond weak signal after a reasonable duration.
- The two strategies were too similar to create a measurable difference.
After Applying a Winner
After the winning strategy is active:
- Check the collection's product preview and live storefront.
- Watch the Analytics dashboard for revenue, orders, CTR, add-to-cart rate, and sort runs.
- Document what worked so future collections can start from better defaults.
- Re-test periodically as catalog mix, seasonality, inventory, and shopper behavior change.
Next Steps
- Creating Tests - Start another experiment.
- Advanced Mode - Build stronger challenger strategies.
- Analytics Metrics - Understand the metrics tests and dashboards share.