Data-Driven Design
Conducted qualitative research to generate insights that defined quantitative data collection which informed design choices.
Context
StarCraft is a real-time strategy computer game, but to call it merely a game does not do it justice.
After it’s release in 1998, StarCraft’s popularity created an enormous grassroots market for South Korea's new broadband infrastructure, the explosion of PC bangs (internet cafés), providing a much-needed microeconomic catalyst that helped lift Korea out of its recession following the 1997 financial crisis.
Goal
Create a ranking system for StarCraft 2 which would drive long-term player engagement and meet the world’s high expectations of industry-leading design excellence.
Problem Statement
We needed to balance competitiveness and accessibility to ensure the ranking system motivated all player types and minimized churn.
Collaborative Analysis
This project was driven by a cross-functional team, including:
Data Scientists (2): Focused on data pipeline to collect data for analysis.
Doctor of Mathematics, Ph.D.: Applied deep expertise in probability and curve functions.
Developers (3): Implemented the required instrumentation for data tracking and deploying the final design features.
Myself: Conducted qualitative research, formulated the core hypotheses, led collaborative analysis to interpret the quantitative data, and drove the final design.
Seven Tiers
The ranking system consisted of seven progressive tiers:
Bronze, Silver, Gold, Platinum, Diamond, Master, and Grandmaster.
Group by Population
Design
The first five tiers were evenly distributed by population (20%), while Master and Grandmaster totaled the top 3%.
User Feedback
Players appreciated the fairness of the population distribution and found it intuitive.
Each tier was divided equally by population.
Group by Skill
Data
Forum feedback highlighted which players were frustrated, and analysis of their MMR showed most were in Bronze or Diamond. Bronze players struggled to progress, while Diamond players were promoted to that tier and felt they had reached the end, knowing they were unlikely to advance to Master despite strong performance. The data showed that both groups were actually improving their skills significantly.
Design
I concluded that grouping by population was causing the issue and that organizing tiers by skill would be more effective.
I reduced the population in Bronze and Diamond while increasing it for Silver, Gold, and Platinum with Gold being increased dramatically.
Continuous Feedback and Iteration
User Research
As time went on, we continued to gather feedback on the website forums and noticed that some players still felt they were getting stuck. Analysis showed that these concerns were primarily coming from Bronze and Silver players.
Data
Analysis of the data revealed that Bronze players were improving their skills significantly but were not receiving meaningful rewards. In contrast, Platinum and Diamond players were experiencing similar skill growth but were highly satisfied with the system.
Personas
Based on these insights and additional analysis, we divided the player population into five data-driven personas.
Bronze / Silver
Who: Value recognition for winning even a single match.
Findings: High drop-off. Progress feels slow, and tiers often perceived as discouraging compared to Gold.
Thoughts: Design should focus on sustaining engagement.
Gold
Who: Players at the middle of the bell curve.
Findings: Players in Gold are happy with their tier placement and feel competent at the game.
Thoughts: Progression indicators should be intentionally vague to sustain positive self-perception.
Platinum / Diamond
Who: Players who value progression in skill.
Findings: Highly engaged. Sense of accomplishment when progression occurs.
Thoughts: Design should reinforce progression milestones.
Master / Grandmaster
Who: The top 5% of players, highly competitive and focused on skill mastery.
Findings: Value accuracy of their own ranks.
Thoughts: Engagement for this group depends on features that reflect true skill.
Groups by Data-Driven Design
Rather than grouping players strictly by population or skill, we grouped them by persona.
The initial design had the population distributed equally across skill levels.
The second design had skill distributed evenly across the population.
The final design had a decrease in Bronze, an increase in Silver, and a large lopsided increase in Gold.
Ranking System by Data-Driven Design
The Bronze, Silver, and lower Gold tiers changed to progression-based system, allowing players to advance even with a win rate below 50 percent.
Movement through the system depended entirely on skill.
Movement in the early tiers depended purely on number of wins while the upper tiers remained skill based.
The animation players see when moving up tiers.
Results
Compared to the last major release, we saw substantial benefits across the board due to this change and others, such as matchmaking, which were based on data-driven design.
+40%
User retention in Ranked Play
+500%
User engagement in Ranked Play
88%
Metacritic score