The last two posts in this series covered measuring churn, both in businesses with a subscription-based business model and in ones where there is no easy way to define “churn.” This blog post will detail how to tune an existing model to give more accurate results.
When building any Classification or Predictive model, there are always multiple iterations – throughout each, we “tune” the model based on its performance on training data. All of the steps that we take, from segmenting the data, to feature engineering, building the model, and then evaluating the model, are typically repeated several times. For our churn model, it is critical to retrospectively evaluate model performance for each phase of development and identify things that can be modified to improve model accuracy.
The accuracy of a model is simply the ratio of the correct predictions to the total number of cases that have been evaluated. However, to improve model accuracy we need to find where the model missed and why it missed – did it predict someone would churn, but they did not? Or, more importantly for our model, did it predict that someone would not churn, but they did? By rigorously interrogating our model, the data will tell us the missing parts of the story and suggest ways to improve our model.
In churn modeling the first thing we need to check is the misclassified customers, specifically the ones that we were not able to “catch” before they churned. These cases are critical since the purpose of predicting churn is to have this information while there’s still time to do something – customer retention is easier and less costly for a business than the acquisition of new customers.
When we circle back and evaluate how we built the model, we need to look for the unseen – the reason underneath the churning of those customers the model predicted to have a low probability to churn. Performing cluster analysis (e.g. K-Means, Hierarchical Clustering, DBSCAN, etc.) will help us find common patterns for this customer group. It is likely that we will see at least some of the reasons they churned. For example, they might be “irregular” buyers with orders placed in wide and unpredictable time intervals. Therefore, either the mean frequency (e.g. mean frequency = customer buys every 45 days) alone might not be enough to identify the churning point, or it should be calculated differently for these types of customers. Another scenario might be that these customers had bad product experiences or unsatisfactory service. For example, if a group of customers who bought the same product churned, it is likely that customers who recently bought it could be at high risk of churning regardless of their normal buying frequency. Including the data that captures this information into the model will improve the quality of the model.
On the other hand, there are customers that were predicted to leave but didn’t. Although the previous ones are more important to prevent revenue loss, these customers are indeed loyal to the company. Retaining loyal customers is essential. Therefore, a business would not want to target them with aggressive campaigns or too many emails. It would be a poor allocation of resources and even worse, these customers might not like frequent contact at that point, and choose to disengage (e.g. not purchase) in the future. Consequently, loyal customers might need a more flexible calculation for their buying frequency that is leveraged by our churn model.
As we see, there are various aspects to be considered when we investigate customer purchase behavior. While much of this information is noise and complex models should be avoided, we must capture the most important elements in the churn definition.
All these kinds of findings are useful input for model improvement. We go back to the very first step to modify the definition of churn for each customer and assign the zero and one labels again. Then, as in the first time around, we train and evaluate the best model. Finally, we conclude that constant observation and improvement is the key to learn and predict customers’ behavior better. The more we know the higher is the probability of building a powerful predictive model.
This is part 3 of 3 in the Mechanics of Predicting Customer Churn series.
Andrew Malinow, PhD, leads the Data Science team at Zylotech, where he leverages his background as a Cognitive Psychologist, statistical expertise and passion for surfacing actionable insights from large, messy data sets. At home he loves to spend time with his wife and 4 kids, doing anything outdoors, and tending to his ever-growing flock of chickens on his farm in Pomfret, CT.
Mimoza Marko is a Data Scientist at Zylotech, where she brings an extensive background in mathematics, statistics and computer science. She is passionate in exploring the mysteries in data. While quality time with family and friends is her favorite part of the day, Mimoza loves painting, hiking, reading and learning about the universe.
If you liked this post, check out our other blog on how data quality can affect your campaign.