I decided to post some churn model outcomes after reading a post by the enigmatic Zyxo on his (or maybe her :)) blog ;http://zyxo.wordpress.com/2009/08/29/data-mining-for-marketing-campaigns-interpretation-of-lift/
I'd like to know if the models rate well :)
I'd love to see reports of the performance of any predictive classification models (anything like churn models) you've been working on, but I realise that is unlikely... For like-minded data miners a simple lift chart might suffice.
The availability of data will greatly influence your ability to identify and predict churn (for the purpose of this post churn is defined as when good fare paying customers voluntarily leave). In this case churn outcome incidence is approx 0.5% per month, where the total population shown in each chart is a few million.
Below are two pictures of recent churn model Lift charts I built. Both models use the previous three months call summary data and the previous month's social group analysis data to predict a churn event occurring in the subsequent month. Models are validated against real unseen historical data.
I'm assuming you know what a lift chart is. Basically, it shows the magnitude increase in the proportions of your target outcome (in this case churn) within small sub-groups of your total population. Sub-groups are rank/sorted by propensity. For example, in the first chart we obtain 10 times more churn in the top 1% of our customers we suspected of churning using the predictive model.
The first model is built for a customer base of prepaid (purchase recharge credit prior to use) mobile customers, where the main sources of data are usage and social network analysis.
The second model is postpaid (usage is subsequently billed to customer) mobile customers, where contract information and billing are additionally available. Obviously contracts commit customers for specified periods of time, so act as very 'predictive' inputs for any model.
- first churn model lift
- second churn model lift
Both charts show our model lift in blue and the best possible result in dotted red. For the first model we are obtaining a lift of approximately 6 or 7 for the top 5% population (where the best possibly outcome would be 20 (eg. (100 / 5) = 20).
The second model is significantly better, with our model able to obtain a lift of approximately 10 for the top 5% of population (half way to perfection :)
I mention lift at 5% population because this gives us the reasonable mailing size and catches a large number of subsequent churners.
Obviously I can't discuss the analysis itself in any depth. I'm just curious what the first impressions are of the lift. I think its good, but I could be delusional! And just to confirm, it is real and validated against unseen data.
- enjoy!
18 comments:
Hi Tim,
I love the title of your post. It is indeed a problem to get to see real data of other data miners.
About your model. I just finished (today) a targeting model for selling a particular investment product (I'm in banking, you know).
Just like you I predict what 's going to happen the next month.
Numbers are comparable to yours : a bit under 0.5% are buyers among a small handful of millions of customers. My 0,5% lift is about 25, my 0,1% lift is about 45 (calculated on unseen data). Compared to my other models it is a tiny bit below average.
So I think the qualities of our models are roughly comparable. And as I am convinced I do a good job myself, you cannot be too bad either :-)
Hi Tim and Zyxo, it is "easy" to get high lift when you have a such rare event (0.5% monthly) , I get much less lift on prepaid churn models because our prepaid has an annualized churn rate of 40% on a couple of million customers :) So I use several other methos of model comparison but always benchmarking on the same population.
AUC figures for churn, upselling, and appetency models in telecommunications here:
http://www.kddcup-orange.com/index.php
Best AUC values are around 76.5%, 90% and 88%, respectively. The winning teams have presented their methodology
Mind you, this is a maths only exercise - anonymized data means you can apply business understanding. With that, we are usually aboe to get close to 80%.
Predicting churn is easy. Deciding what to do about it is hard. More thoughts on this at
http://www.cybaea.net/Blogs/Journal/KDD-Cup-2009.html
Hi Allan,
my AUCs are in the range of 78-81%, but I agree that finding a profitable - or at least a non-value destroying means of keeping the customers is difficult but not impossible. I did a 6 months test - 3 months treatment and 3 months extra measurement and found that it is possible to keep some of the customers profitably.
Sorry for the misprint because I misread my calcs :) My AUC's cummulated are around 66%...
Pardon, my Gini index is around 66%...I need to get a cup of coffee before making comments so early in the day :) Sorry for the multiple replies :)
Thanks for your comment guys.
Couple of replies;
a) Hello Ivan,
I'm curious to know what your definition of churn is. I assume that because you are quoting 40% that it is simply any customer that was here one month and gone the next. This might include deceased customers, holiday makers, and other transients that no marketing action could ever prevent.
My definition of churn is port-out; where a customer transfers their number to a competitor (we therefore have an IT mechanism that tells us a fare paying customer has gone to a competitor) and there is obviously loss of future revenue. If we can confirm that a marketing campiagn might have a chance of retaining that customer, then it is defined as churn. Otherwise it is simpy disconnection (which includes dead people, inactives etc).
Prepaid customers usually become inactive for several months before we simply disconnect them, and we have separate models that predict the onset of this inactity with similar success.
b) Hello Allan,
Completely agree with your post. Once I saw the var1, var2 etc I dismissed the KD Challenge as a joke. I sent an email to the organisers requesting they make it more realistic in future;
http://timmanns.blogspot.com/2009/03/tips-for-kdd-challenge.html
Hi Tim,
our definition of prepaid churn is 3 months without revenue generating activity. Because 90% of our prepaid is unregistered we can't track who ported out and when. We have bounce back effect on some but still in our market churn is high because of saturation (120% SIM penetration) and very competitive environment. That is why we are predicting the event that someone will go into a specific number of inactive days that will with high certainty eventually lead to our system definition of churn (above).
HI Ivan,
I kinda feel sorry for you :) You've a much harder job to predict churn if there is no easily way to measure the outcome. It sounds as though your available data is scarce too.
I've got access to name, address, age, gender and a few other bits of basic information about every customer, even prepaid. Also a weatlh of detailed CDR usage of course. The only exception is where the services is purchased by a parent (on behalf of a child) in which case we only have the personal details of the same parent for both prepaid services.
tim - you seem to have very little stats expertise. does that make your managers nervous? (please correct me if i'm wrong.)
Hi Anonymous,
Mmm, interesting comment. I'll bite.
Expertise is is a relative term :) and my managers aren't nervous. I know what a T-test is, although never used one since I left University. I have built my own neural network from the ground-up, so I guess i know what neural nets are. I don't make sampling errors, and I validate everything against unseen data over a period of a few months, so I believe I sufficiently test my work.
I'm fortunate to have a great data warehouse and lots of rich data to work with that makes creating successful analysis (and models) easier. Every data miner has different preferred methods of working and different data available to work with. These impact the outcomes greatly. The reason for my post was to get some insight into the outcomes my peers might achieve.
If you are suggesting that in order to be a good data miner or create predictive models you must have a phd is statistics then I disagree. Some of the worse data miners I've meet were professors, and stats plan a tiny (but important) part in the process...
Most of the problems I deal with are data manipulation.
Maybe changing the topic, but thought you might like the following quote from Leo Breiman...
"But the truth of the matter is -- which I have never admitted publicly before, that I have never taken a course in statistics in my life. Neither has my friend and colleague Jerry Friedman, who just finished three years as chair of the Stanford Statistics Department. He was an experimental physicist who got his start in statistics by analyzing tracks of high-energy particle collisions in the Stanford Linear Accelerator. John Tukey was a pure mathematician, George Box a chemist. Many other distinguished statisticians somehow drifted onboard. "
In my experience statisticians can be too concerned with t-test etc that they miss the bleedin obvious (I am from machine learing not stats)
Tim - I am abit surprised you dismissed the KDDCup. Are you saying wouldn't have learnt anything from the experience? I certainly did.
Hi Phil,
I was just too frustrated with the KDD challenge for not supplying rich CDR data. The challenge is a good one though and I could have tried something and submitted an entry, but it would have been very different from what I *hoped* to do (and what I typically do at work). I prefer the challenge of dealing with real data rather than the challenge of tweeking models. I have been reading posts on the KDD challenge results, and certainly got some useful info.
looks good tim. if you make these XY plots in Excel i think it will better highlight just how pointed your model is at the 1-5% end...
cheers
shane
Hi Tim
Very interetsing post - I have a question which I need a bit of help with :
RE: "I'm curious to know what your definition of churn is. I assume that because you are quoting 40% that it is simply any customer that was here one month and gone the next. This might include deceased customers, holiday makers, and other transients that no marketing action could ever prevent."
How do you caluclate prepaid churn? isn't a prepaid churn just someone who stops using the service liek Ivan says or do you use additional filters i.e. removing travlers etc?
Hi Shane and Justin,
- Shane. I intentionally didn't want too much detail in the chart. At 1% of population we at at approx 15 lift, and at 5% population we are at approx 10 lift. After discussions with Dean Abbott last week at PAW I have a few ideas I will be trying out that might improve this further. Time will tell :)
- Justin. You are correct that my definition of churn is far more selective. Only where I can prove;
a) voluntary churn and
b)loss of future revenue loss
- only then is a lost customer considered churn. In my case I use port-out as that definition. Port-out is an IT process where the customer transfers their phone number to a competitor. In Australia is is very easy to do and it is openly advertised and a popular customer action. For obvious reasons I can't say exactly what the Optus port-out number or % is (it is market sensitive data). As an example assume it is 0.5% per month.
It appears to me that by having a good (and accurate) defintion makes predicting that strict outcome much easier. I think Ivan has a more difficult job than me because he has no certain way to define churn from holiday makers, deceased customers etc. These outcomes are really 'noise' and make accurate prediction more difficult. If possible, having a strict definition of churn will enable you to get informative insights and models.
Also bare in mind the *use* of that predictive model. We want to remove deceased customers and holiday makers because no marketing activity or wasted retention cost wil be able to change their behaviour and retain them as customers.
Regarding someone who just stops using the service, we define this as on-set of inactivity. We do also have predictive models that attempt to target these customers one month before on-set of inactivity. Obviously this is a far more general definition of churn, and like Ivan, our models have a 'weaker' ability to predict this general outcome. I personally don't like the exercise of predicting inactivity because there are too many unknowns, but we do it and the results are good (although not quite as good as predicting port-out).
Good one.You might be interested to take a look at the collection of Tutorials and videos on Data mining.
Tutorials: http://www.dataminingtools.net/browsetutorials.php
Videos: http://www.dataminingtools.net/browse.php
Glad to see humor is still alive and well in this data driven world. Funny title.
Post a Comment