Customer Data & Analytics Blog

Use Case: Customer data to support A/B testing

Josh Fayer | 2 minute read


A/B testing is an incredibly powerful and widely-used methodology to figure out what resonates with your customers. At a basic level, A/B testing is a statistical trial of sorts that compares two subjects in real-time. Does layout B increase time on our website? Does button color A or button color B generate more clicks? These questions and others can be used to help compile some comparative metrics for your team. Because the two trials are run concurrently and on randomized populations, marketing and data science teams can control for other variables that might affect the results.

CDPs already allow for extensive segmentation of customers. Within pre-built customer segments, you can randomly create subsegments and compare the effectiveness of a given campaign to observe how different populations respond to stimuli.

An example might make this clearer. Imagine you have a group of customers who you know respond well to sales. You decide to do a promotion, so this group makes the most sense to target rather than wasting resources targeting your whole database. Intuitively, to maximize profits from the sale, you need to figure out what price point will drive the most customers to buy while keeping the price as high as possible. This is something that can be hard to work out with best guesses.

Take, for example, a retail store that is pushing out a $100 jacket. They’re willing to give as much as 50 percent off the price, but the end goal of a sale is to generate as much revenue as possible. Will an equal number of people buy the jacket at 40 percent off compared to the 50 percent off group? In order to break even, they would need 6 people to buy at 50 percent off for every 5 people who buy at 40 percent off. Otherwise, they’re throwing away $10 in profit on every jacket. In other words, they need to figure out if a marginal discount will really generate 1.2x the number of conversions.

The retail store can take their CDP segment for customers who respond well to sales, and randomly split it in half. They can assign one half a sale of 40 percent off, and the other half a sale of 50 percent off, and observe conclusively which sale generates more revenue. With enough similar trials of sufficient scale, that brand can learn exactly what kinds of sales perform best.

Marketing teams can do this for almost any type of change. New website UIs, new sales, new catalogs or newsletters, on and on. CDPs can help brands find these starting points to generate new insights about their customers. Like other campaigns, A/B tests often aren’t free to operate. It’s a waste of resources to send a sale to someone who doesn’t respond well to sales at all—running A/B tests on highly specific and accurate groups is the best way to ensure you’re testing on the customers that will most likely generate solid revenue streams for your business.

Josh Fayer manages Marketing Communications at Zylotech, where he brings an extensive background in public relations and computer science. Josh Loves music! He sings with an a cappella group, Orange Appeal, (whose recent album Unpeeled he shamelessly plugs at work), and at home he loves to mess around with a variety of instruments. Josh enjoys spending quality time with his adorable dog, and exploring around his new home near the South Shore of Boston.

If you liked this post, check out our other blog post about how AI can benefit your business.

Topics: Customer Intelligence