Test Results Not Yet Statistically Significant
While the mobile segment shows 95% confidence (+27% CR), the overall test results are not yet statistically significant. We recommend running the test longer to validate these findings before making permanent changes. The current data is promising but requires more time to reach conclusive significance across all segments.
We tested a new mobile menu that prioritizes flavors
By showing customers the flavor list upfront with "Build Your Own Bundle" highlighted, we saw a +27% conversion rate increase on mobile with 95% statistical confidence. However, average order value dropped 14%, revealing opportunities for bundling optimization.
Original menu with standard navigation
Flavor-first menu design
Executive Summary
Key Findings
- ✓+27% mobile conversion rate with 95% statistical confidence
- ✗-14% average order value across all devices
- ⚠Desktop performance unclear - not statistically significant yet
- ✓Customers buy 2.2 products per order - opportunity for bundling
Top 3 Recommendations
- 1Roll out mobile-only - Capture the +27% CR win while iterating on desktop
- 2Install heat maps - Understand why AOV dropped 14%
- 3Optimize product pages - Test Javvy/Fum patterns to increase multi-unit purchases
The Hypothesis
What We Changed
- ✓Moved "Build Your Own Bundle" to the top with prominent red banner
- ✓Displayed all flavor options directly in the menu (Pink Lemonade, Grapefruit, Chocolate, Orange, Unflavored)
- ✓Moved secondary links (Build a Bundle, Merch, About Us, etc.) to the bottom
Why We Tested This
We believed that by showing customers the flavor options immediately, they would find their preferred flavor faster and proceed directly to purchase. The hypothesis was that reducing navigation friction would increase conversion rates by letting customers self-select based on taste preference rather than browsing through multiple pages.
Video Walkthrough
Watch this 12-minute video for a detailed explanation of the test results and recommendations
Key Results
Overall Test Performance
The Real Story: Mobile vs Desktop
When we segment by device type, we see dramatically different results. The new menu performs exceptionally well on mobile but has minimal impact on desktop.
Detailed Breakdown by Device
Statistical Confidence
Using AB Test Guide's calculator, we can confirm that the mobile results are statistically significant at 95% confidence level. This means we can be confident the improvement is real and not due to random chance.
Variation B's observed conversion rate (4.15%) was 27.27% higher than variation A's conversion rate (3.26%). You can be 95% confident that this result is a consequence of the changes you made and not a result of random chance.
What We Learned About Your Customers
Beyond the test results, we analyzed your order data to understand typical customer behavior. These insights reveal opportunities to increase average order value.
Your average order value is around $60 during normal periods (vs $98 during Black Friday). This represents your baseline to improve from.
On average, customers buy 2.2 products per order. Most commonly, they purchase 2 units of the same flavor rather than mixing flavors. This shows an opportunity to encourage larger same-flavor purchases.
Recommended Next Steps
Based on these results, here's what we recommend testing next to maximize both conversion rate and average order value.
Roll Out New Menu on Mobile Only
Since the new menu has a clear positive impact on mobile (+27% CR with 95% confidence) but performs poorly on desktop, we recommend implementing it exclusively for mobile users. This gives you an immediate conversion rate boost on your highest-traffic segment while we iterate on the desktop experience.
Install Heat Maps to Understand the AOV Drop
We need to understand exactly why AOV dropped 14%. Heat maps will show us where customers click in the new menu. For example, if they're clicking "See All Flavors" and then getting disappointed by limited options, we should show all flavors upfront. If they're ignoring the bundle option, we need to make it more prominent.
Optimize Product Pages & Bundle Builders for Higher AOV
Since customers already buy an average of 2.2 products per order (usually 2x the same flavor), we should make it easier and more rewarding to add multiple units. The goal is to increase AOV from $60 to $75-80 by encouraging customers to buy 3+ units instead of 2. We need to test both individual product page improvements AND bundle builder approaches.
A. Product Page Improvements
Test different quantity selector patterns that make multi-unit purchases more visible and rewarding:
Current product page requires significant scrolling before reaching the Add to Cart button. No visible incentive to add multiple units.
View live page →
Javvy shows a progress bar that gamifies adding multiple bottles. Clear pricing tiers (1, 2, 3, 4+ bottles) with "BEST DEAL" badge make the value obvious.
View live page →
Fum auto-adjusts pack size as customers add items (3 Pack, 6 Pack, Complete Pack). No need to choose quantity upfront. Shows subscription option and upsells accessories.
View live page →Ready to Move Forward?
Let's discuss how we can implement these recommendations and continue optimizing your store. Book a call to explore if we're a good fit.
Or email us directly at [email protected]
Summary
The new mobile menu is promising but needs refinement. We have a clear win on mobile (+27% CR) that we should roll out immediately, but the AOV drop (-14%) needs to be addressed through better bundling incentives and product page optimization. With heat maps and iterative testing, we can capture the conversion rate uplift while protecting (or even improving) average order value.