The challenge
I have been an analyst for over 12 years at numerous companies, both inhouse and as an external consultant. And all these companies have something in common.
Whenever there was a significant dip in a business critical KPI everybody had to drop whatever they were doing and needed to search for, what seemed to me, a needle in the haystack.
When the dust settled I always thought this situation could have been handled differently.
It’s very resource intensive and is incredibly cost inefficient.
Causal inference
The challenge is that many factors can influence outcomes, and sometimes these factors can trick us into making incorrect conclusions.
Some years back I stumbled across causal inference.
Cause inference is the study of determining whether changes in one thing lead to changes in another, distinguishing correlation from causation in observational data.
Imagine you see rain pouring outside, and then you notice people carrying umbrellas. You might think, "Rain causes people to carry umbrellas."
But is it correct to conclude that?
This is where causal inference helps us to sort through the data to make more accurate conclusions about cause and effect.
Basically, determining if rain is indeed the main cause of people carrying umbrellas.
This seemed like a good direction to help me solve the challenge I just described, but how could I apply causal inference?
I found an article from AWS called “Root Cause Analysis with DoWhy, an Open Source Python Library for Causal Machine Learning”, which I recommend to everybody invested in the topic.
This blog post is basically a blueprint on how to efficiently approach our challenge.
DoWhy and how it can help
So DoWhy is an open-source Python package and companies like AWS are contributing to it’s development. It helps analysts identify the root causes of observed changes in complex systems. And it can do so in an automated way. Sounds like something we could use right?
The model starts off with defining a cause and effect diagram as input for our model.
Then it will use this cause and effect diagram as a basis to identify relationships and the strengths of the components within that diagram.
To make this more visual I’ll use an example.
Real world example
Imagine you're an email marketer for an e-commerce company. You're responsible for the e-mail marketing channel.
To keep track of your performance you’re monitoring the following KPI’s:
- Email revenue share
- Email turnover
- Email related
- Transactions
- AOV
- Sessions
- Conv.ratio
- Sends
- CTR
- Subscribers
- Frequency
- Existing subscribers
- Unsubscribes
- New subscribers
You can say that all these KPIs are related to each other and one affects the other. But did you know you can leverage that principle to your advantage?
Step 1: Casual Diagram
Instead of just summing the KPI’s up you can also frame these in a causal diagram to see the direct and indirect relationships between the KPIs. See an example below:
Step 2: Feed and train your model with data
After you have defined the causal diagram you can give the model your data to learn and understand the relationships and strength.
The dataset will basically hold all the performance of the KPI’s but on a daily level. Below a partial example:
DateUnsubscribesSubscribesSendsKPI etc..202308121030100000…20230813510090000…Example model data
Step 3: Ask the model for insights
Now that the model is trained you can ask the stuff that really matters.
It's Monday morning. And it happened. Email revenue dropped last weekend and its absolute mayhem. And it’s you, the email marketer, people expect answers from.
The revenue chart can look something like this:
And while everybody is panicking you can find the cause based on data by asking the model:
Dear model, What are the key factors explaining the email revenue drop on a Sunday?
And the model gives you these factors on a silver platter:
How awesome is that?
The graph above displays all the KPIs in the causal diagram you defined and gives them an “Anomaly Attribution Score”. More details about the interpretation of the score can be found in this research paper. Simply put the “Anomaly Attribution Score” tells you to what extent the KPI contributed to the effect and to what extent it did not.
This is not a direct answer to what caused the revenue drop but it does give you a more precise location into where to look, which saves massive amounts of time.
Going above and beyond
DoWhy can do a lot more than I just showcased. You can include external and internal factors such as extra data attributes.
DateUnsubscribesSubscribesHot summer dayDiscount period202308121030TrueTrue202308135100FalseFalseDoWhy data example
Was it a hot summer day? Did you have a discount period?
For every data point or variable you feed into the model, it can tell you to what extent those variables have an effect on the trend change.
Things to keep in mind
A causal model is only as good as the input the user gives it. And the model relies on certain assumptions.
Let’s explain with an example.
Imagine you're investigating whether the "Awesome Discount Promotion" email boosts sales for an e-commerce company. You have to check if the following assumptions are met.
Assumption 1: Stable Unit Treatment Value Assumption (SUTVA)
- No interference: If Bob gets the Awesome Discount email and makes a purchase, it doesn't mean Alice, who also received the email, will buy something because of Bob's purchase. Everyone's buying decisions are independent.
- Consistency: If Alice buys a product after receiving the Awesome Discount email, it's because of the email promotion, not because she had a random desire to shop at that exact moment.
Assumption 2 Positivity or Overlap
Among the customers we're studying, there's a chance any single one might receive the Awesome Discount email… or not. It's not like only new customers get the promotion and loyal customers don’t. We need a mix of people to make a good comparison.
Assumption 3 No Hidden Bias
Think of this as making sure there's no secret event influencing sales. For example, if you consider factors like previous purchase habits and how often people visit the site, this assumption means there's no hidden factor, like a viral product video, influencing sales at the same time as your email promotion.
Assumption 4 Monotonicity
Let's say the reason some people received the Awesome Discount email was due to a social media influencer mentioning the promotion. This assumption would mean that the influencer's mention either encourages people to look for the promotional email (or has no effect) but doesn’t make anyone avoid or delete the email without reading.
Using these "rules of the game," we can fairly evaluate if the "Awesome Discount Promotion" email genuinely boosts sales or if other factors are at play.
Of course it's unreasonable to always meet all of these assumptions but this will show you the boundaries and limits of your model.
Going back to our email marketing example DoWhy will definitely point you in where to look but it’s likely that you don't have access to the underlying factors that fully explain the drop. E.g. best friends telling your customers to not open certain commercial emails.
Unfortunately, that’s the reality.
That’s a lot to take in, but you made it this far, so you’re probably asking: ok, so how does this work practically?
https://www.youtube.com/watch?v=Hmeis9C_TGA
I hope this gives you an idea how this would work day-to-day.
At SEINō we are currently investigating if such a model would also be beneficial for our current and future customers. We really would love to hear your thoughts.
Please feel free to reach out and share your take!
Sources:
https://www.optmyzr.com/blog/optmyzr-automates-ppc-account-audits-with-ppc-investigator/
