At first glance, it can be intriguing to solve an isolated need, such as getting more newsletter subscribers, by simply using a DIY-tool and activating a simple pop-up. In most cases, it will certainly help you collect more subscribers. However, you may not be aware of the hidden negative effects that come along with this type of solution.

  • Text Hover
  • Text Hover


There’s a pretty easy way to identify and fix these issues.

For the majority of our clients, the realization that a DIY pop-up service can have a negative impact on a variety of KPIs – such as revenue – has been a real eye-opener. The issue will often boil down to the one-size-fits-all approach of these types of services.

Some website visitors enter a site with a clear goal in mind, or even with a solid purchase intent, only to be distracted by a pop-up asking them to subscribe to a newsletter or visit another area of the website. In this case, an inappropriately timed pop-up is comparable to a store clerk physically removing a product from a customer’s shopping cart. In other words – you’re losing sales.


However, not all visitors are at the same place in their customer journey, and as such, will react differently to the same pop-up communication. Our analysis shows that factors such as traffic source or what device and browser you’re using can have a major impact on how much revenue you are losing (or gaining).

Addressing this issue is not rocket science, but something that every e-commerce brand needs to address and be aware of. Below are our 4 steps toward source and device analysis.


To start this analysis, you need to set up a split test where a portion of your target group will remain as a control group. In other words, they won’t see your overlay, even though they fulfill your segmentation criteria. This will allow you to compare those who have seen your overlay with the control group, giving you valid data to analyze. When you have collected enough data to provide a statistically valid result, you’re ready to move on to the next action (when exactly this is, will depend on your traffic levels, segment size, and results).


Once you have your data set ready, you need to set up two cross-tabulations focusing on your key metrics. To be specific, you should extract session, transaction, and revenue data from your relevant segments in Google Analytics (or similar analytics system) split out by traffic source as well as device type. One table will focus on the visitors who were exposed to your overlay, and the other will highlight data from your control group.


After collecting and inputting your data in your two tables, you are ready to compare your key metrics. Typical KPIs to focus on include differences in conversion rates (CVR), total sales (comparatively), and average order value (AOV). In certain cases, it may also make sense to analyze additional engagement parameters, such as how time on site or pages per visit are affected. Most of us are aware that user experience and behavior is greatly impacted by which device we are using, but what is more surprising, is the difference in performance when comparing traffic sources.


So what's next? At this point you've collected and analyzed your data, providing you with some insight on which types of users should see your pop-ups - and who definitely shouldn't. Keep in mind that these conclusions should typically not be made based on a negative CVR impact alone, but should also take into consideration how showing an overlay impacts factors such as AOV. Based on your conclusions, you can now begin to adjust your segmentation and trigger rules, to accommodate your new learnings. Keep in mind that this should not be a one-time analysis, but rather a regular exercise that should be a part of your website optimization strategy.


For most e-commerce businesses, this analysis will show you where you need to optimize in order to reduce the negative effects of an unsegmented pop-up strategy. It also highlights the major benefits of working with a data-driven partner that focuses on your entire business, and not just one isolated KPI (such as sign-up rate).

All in all, these 4 steps should help you identify potential and help you prevent your pop-ups from stealing your revenue.

If you would like to hear more about our approach to optimization and data-driven development, please get in touch.