Joseph Kim, Founder of blog Game Makers discusses root cause analysis, highlighting the value of different thought processes for game product managers to consider when problem solving in order to optimize game design.
Read on for edited highlights from Kim's podcast:
What is root cause analysis?
“The fundamental thing with root cause analysis is just trying to find the root cause of a problem. It’s trying to solve issues at the source level by first identifying the key issues. Don’t focus on the symptom, find the underlying issue with the game.
Basically, it’s a very fundamental approach to try to identify what the core issue is that’s causing problems such as poor CPIs and monetization retention. This is accomplished through various types of thinking analysis tools.”
The motivation behind using root cause analysis for optimization issues
“For live-ops and optimization, root cause analysis often becomes vitally important. In terms of motivation, the type of products and issues we’re confronted with as product managers in free-to-play games are so complex, involving so many variables, that I think it’s important to understand whether the work you’re doing is actually in the right direction. That’s what root cause analysis can help you do.”
Are there different approaches to root cause analysis?
“People think doing root cause analysis is just about learning a specific methodology or just another form of analysis. But there are actually other aspects that can impact how you do root cause analysis that aren’t often talked about - like how you should actually be thinking and the work environment that you are operating in. These conditions could impact how you execute root cause analysis.”
Key factors of company culture for efficient problem solving
“The main issue is having a culture that is open. Let’s say that you’ve just launched a game and you’ve got specific revenue targets set for that game. Now let’s say leadership comes to you asking for a presentation that explains why the game is performing poorly and how to fix it by the end of the day. Well, in that scenario, it kind of forces a product manager to go into a way of thinking that may not be the right way.
In certain large organizations, you’re not able to identify a specific person or cause of a problem because it just wouldn’t be politically correct to do that. So I do think that being in the right environment and having the right kind of culture that allows you to do the analysis correctly is often problematic.”
Culture as a live product
“Culture needs to be constantly iterated and approved upon and watched. It is a little bit more of a risk with larger and older organizations as the processes and the way of thinking become outdated. When you’re a startup and when you’re a lot smaller, you’re survival is on the line, so there’s a much greater willingness to try new things and to be more open. If you’re not, you’re going to die.”
3 different types of thinking
1. Thinking fast and slow
2. Reductionist and holistic
3. Global vs local
Thinking fast and slow
“This concept actually comes from a book by Daniel Kahneman who argues that there are two types of thinking. The first he calls System 1 and the second he calls System 2. System 1 is an automatic, fast, unconscious way of thinking. System 2 is the more effortful, slow, controlled, and deliberate type of thinking.
So, if I ask you what is 2+2, your brain would kick off the System 1 type of thinking. If I ask you what is 13 x 17, then you’ve got to spend a little more time and your brain would start using System 2.
The point that Daniel Kahneman makes is that different kinds of problems require different types of thinking. While this may seem obvious, you’d be shocked by the number of people who actually don’t do deep thinking against critical problems. There are a lot of what I call ‘check-list organizations’ - meaning organizations that are just about ‘Okay we need to do X, Y and Z. Check, check, check.’ If there’s a task or issue about really thinking deeply against a critical problem, there’s not a full-force effort.”
Reductionist and holistic
“Reductionism is how you think of breaking down complex problems. In western culture especially, there’s bias toward reductionist thinking. The typical western reductionist approach is to break down complexity by sub components and address that complexity through division. But you have to have different kinds of thinking and not just this analytical component-based approach, which may not be the best approach against all classes of problems.
Let’s say that you’re walking through a park, and you see this really beautiful flower. If you decide to take it home, however, that flower is going to die. Why? Because it needs it’s environment filled with soil and sun in order to live. One of the things that often happens with this reductionist, decomposition-based approach is that it doesn’t take the environment into context. Time and time again we’re confronted with issues and problems where, for lack of a better analogy, product managers are taking this flower home and then they’re scratching their heads wondering why the flower dies. You have to have the full context - the holistic part of thinking.”
Global vs local
“This issue was popularized by the Silicon Valley blogger now-venture-capitalist Andrew Chen about how optimization leads to incremental improvement. Product managers could be optimizing and making incremental improvements, but might be sitting at a local maxima. There may be some other global maxima elsewhere that can lead to much more dramatic improvements in performance.
The caution that I would give to a lot of product managers is to be careful about optimizing against a local maxima and not pushing enough so that they’re able to achieve much bigger global maxima, thinking bigger. Fortnite is a good example. The game was failing but because of Battle Royale and Battle Pass, Fortnite wound up finding much bigger global maxima from a gaming play retention and monetization perspective.”
How to know when you’ve arrived at the root cause
“Deciding whether you’ve done enough is definitely the most difficult aspect, but one way to address this is through a structured approach.
Once all the ideas are out there, decide whether or not those ideas will make a big enough change. Then characterize the cost and the risk of each of the things you’re trying.
To some degree, it also depends on your environment and your strategic context because there are impractical reality limitations. Fundamentally, there’s a meraged, structured way of thinking and modeling as well as the gut feel that just tells you that this new feature is going to cause a big uptick in performance, even though you can’t prove it through data.”
What about the science-side of cracking the problem? Common methodologies
“The most common approach for product managers is just looking at popular KPIs and wildly guess. One level of sophistication up from that approach is what I call KPI decomposition. In this type of analysis you would not only look at key KPIs but then you would decompose those KPIs into parts. This is actually a very reductionist approach as you would continuously break that down until you can identify a specific issue to focus on.
There’s also hypothesis-driven analysis and a type of analysis which I call ‘playbook analysis.’ The latter is essentially when there’s a common list of key problems that different teams have experienced on past projects as well as the solutions that were used. The analysis is documented, so people can go back and learn from what people have tried in the past.
The last common methodology used is logic trees. Generally, there are two types of logic trees: an issue tree and a hypothesis tree. An issue tree is used when we’re trying to uncover a lot of potential ideas and discover the ‘how’ question to achieve a desired outcome. Differently, the hypothesis tree highlights a specific line of investigation and focuses on the ‘why’ question of believing whether something is true.”
The ideal background for today’s game product managers
“The best background for a product manager depends on the situational context, such as the kind of product you’re working on or how the team is structured. The culture of the team that you are part of and whether or not you’re a good match for that environment is also crucial. Generally, a great product manager is one that has...
1. A strong quantitative background
2. Great logical and structured thinking
3. The ability to use different types of thinking and awareness of cognitive and political biases
It’s important for a product manager to focus on continuous improvement and learning. A lot of organizations tend to focus on product when thinking about continuous improvement. For me, the best product managers are those that not only focus on improving the product but also on the organization of the team. The ideal product manager treats their teams and their organizations like a product in need of continuous improvement as well as the product itself.”
Want to learn what goes into making the world’s most popular games? Subscribe to LevelUp to never miss an update.
