Data-Driven Design
Conducted qualitative research to generate insights that defined quantitative data collection which informed design choices.
StarCraft 2
StarCraft 2 is one of the most competitive computer games in the world with professional tournaments offering prize pools of up to $1 million.
Fun Fact: The popularity of the first StarCraft played a microeconomic role in lifting South Korea from its 1997 to 1999 financial crisis.
Goal
Drive long-term player engagement through a competitive ranking system for the real-time strategy game StarCraft 2.
Problem Statement
We needed to balance competitiveness and accessibility to ensure the ranking system motivated all player types and minimized churn.
Collaborative Analysis
This project was driven by a cross-functional team, including:
Data Scientists (2): Focused on data pipeline to collect data for analysis.
Doctor of Mathematics, Ph.D.: Applied deep expertise in probability and curve functions.
Developers (3): Implemented the required instrumentation for data tracking and deploying the final design features.
Myself: Conducted qualitative research, formulated the core hypotheses, led collaborative analysis to interpret the quantitative data, and drove the final design.
Seven Tiers
The ranking system consisted of seven progressive tiers based off skill:
Bronze, Silver, Gold, Platinum, Diamond, Master, and Grandmaster.
Group by Population
Design
The first five tiers were evenly distributed by population (20%), while Master and Grandmaster totaled the top 3%.
User Feedback
Players appreciated the fairness of the population distribution and found it intuitive.
Each tier was divided equally by population.
Group by Skill
Data
Forum feedback highlighted which players were frustrated, and analysis of their MMR showed most were in Bronze or Diamond. Bronze players struggled to progress, while Diamond players were promoted to that tier and felt they had reached the end, knowing they were unlikely to advance to Master despite strong performance. The data showed that both groups were actually improving their skills significantly.
Design
I concluded that grouping by population was causing the issue and that organizing tiers by skill would be more effective.
I reduced the population in Bronze and Diamond while increasing it for Silver, Gold, and Platinum with Gold being increased dramatically.
Continuous Feedback and Iteration
User Research
As time went on, we continued to gather feedback on the website forums and noticed that some players still felt they were getting stuck. Analysis showed that these concerns were primarily coming from Bronze and Silver players.
Data
Analysis of the data revealed that Bronze players were improving their skills significantly but were not receiving meaningful rewards. In contrast, Platinum and Diamond players were experiencing similar skill growth but were highly satisfied with the system.
Personas
Based on these insights and additional analysis, we divided the player population into five data-driven personas.
Master / Grandmaster
Who: The top 5% of players, highly competitive and focused on skill mastery.
Findings: Value accuracy of their own ranks.
Thoughts: Engagement for this group depends on features that reflect true skill.
Platinum / Diamond
Who: Players who value progression in skill.
Findings: Highly engaged. Sense of accomplishment when progression occurs.
Thoughts: Design should reinforce progression milestones.
Gold
Who: Players at the middle of the bell curve.
Findings: Players in Gold are happy with their tier placement and feel competent at the game.
Thoughts: Progression indicators should be intentionally vague to sustain positive self-perception.
Bronze / Silver
Who: Value recognition for winning even a single match.
Findings: High drop-off. Progress feels slow, and tiers often perceived as discouraging compared to Gold.
Thoughts: Design should focus on sustaining engagement.
Group by Data-Driven Design
Rather than grouping players strictly by population or skill, we grouped them by the personas we identified through the data.
The initial design had the population distributed equally across skill levels.
The second design had skill distributed evenly across the population.
The final design had a decrease in Bronze, an increase in Silver, and a lopsided increase in Gold.
Ranking System by Data-Driven Design
The Bronze, Silver, and lower Gold tiers changed to progression-based system, allowing players to advance even with a win rate below 50 percent.
Movement through the system depended entirely on skill.
Movement in the early tiers depended purely on number of wins while the upper tiers remained skill based.
The animation players see when moving up tiers.
Results
Compared to the last major release, we saw substantial benefits across the board due to this change and others, such as matchmaking, which were based on data-driven design.
+40%
User Retention in Ranked Play
+500%
User Engagement in Ranked Play
$9.86 on AliExpress
Guest Speaker at Wharton School of Business
Gamification
I was invited as a guest speaker to the Wharton School of Business by Kevin Werbach, author of “For the Win”, to give a presentation on increasing business success through game thinking, a concept known as gamification.