A/B Testing Overview
Learn how SortLab compares collection sorting strategies with balanced switchback tests on Advanced and Enterprise.
A/B testing lets you compare a collection's current strategy against a challenger before you permanently change the sort order. It is available on Advanced and Enterprise plans.
What SortLab Tests
Each test compares two strategies on the same Shopify collection:
| Variant | Meaning |
|---|---|
| Control (A) | The collection's current active strategy. |
| Challenger (B) | The new strategy you configure during test setup. |
SortLab applies one variant at a time, switches between them on a balanced schedule, and attributes performance to the active variant windows.
Balanced Switchback Testing
SortLab uses a switchback design rather than a simple one-time split. The active strategy changes every 3, 6, or 12 hours depending on your setting.
The schedule is balanced so both variants receive exposure across mornings, evenings, weekdays, and weekends. This reduces time-of-day and day-of-week bias compared with naive A-B-A-B alternation.
Switchback attribution groups orders by product ID and the collection's active variant time window. It is designed for comparing strategy performance, not proving that every buyer viewed the tested collection before purchase.
Metrics Tracked
SortLab reports performance for each variant, including:
| Metric | Why it matters |
|---|---|
| Revenue | Total product revenue attributed to the variant's active windows. |
| Units sold | How many product units moved while the variant was active. |
| Orders | Order count for products attributed to the tested collection. |
| Days active | Exposure time for the variant. |
| Revenue per day | Revenue normalized by active days so variants can be compared fairly. |
| Units per day | Unit velocity normalized by active days. |
| Confidence | How likely the observed difference is real rather than random noise. |
The winner is based on revenue per day when confidence is sufficient.
Prerequisites
Before creating a test:
- Your store must be on Advanced or Enterprise.
- The collection must already have an active SortLab strategy.
- The collection should have enough traffic and orders to produce useful results.
- Avoid major product membership, price, promotion, or inventory changes during the test if possible.
Starter stores see the A/B Tests page as an upgrade prompt.
Test Settings
| Setting | Options |
|---|---|
| Switch interval | Every 3, 6, or 12 hours. Default is 6 hours. |
| Duration | 7, 14, 30, 60, or 90 days. Default is 14 days. |
| Auto-apply winner | When enabled, SortLab can apply the winner after the test completes. |
| Challenger strategy | Any supported preset or custom strategy configuration. |
When To Run a Test
Run A/B tests when:
- You are changing a high-revenue collection.
- Two strategies are plausible and you need evidence.
- You want to validate a custom Advanced Mode strategy.
- You are changing schedule, lookback, new product behavior, or important overrides.
Avoid testing very low-traffic collections first. They may need a long duration before confidence becomes useful.
Next Steps
- Creating Tests - Set up a test from the A/B Tests page.
- Interpreting Results - Read confidence, revenue per day, and winner state.
- Analytics - Monitor collection performance outside tests.