Data-Driven design
Conducted qualitative research to generate insights that defined quantitative data collection which informed design choices.
Role: Lead UX Designer, research and define critical user pain points, guiding a team of six to analyze data and shape design direction
Goal
Drive long-term player engagement beyond launch.
Problem Statements
Player activity typically drops sharply post-launch.
Collaborative Analysis
Our project was driven by a cross-functional team, including:
Data Scientists (2): Focused on data pipeline to collect data for analysis.
Doctor of Mathematics, Ph.D.: Applied deep expertise in probability and curve functions.
Developers (3): Implemented the required instrumentation for data tracking and deploying the final design features.
Myself: Conducted qualitative research, formulated the core hypotheses, led collaborative analysis to interpret the quantitative data, and drove the final design.
DESIGN PROCESS
Research
Hypothesize
Design
Test
QUALATIVE USER RESEARCH
Started with qualitative research to uncover users’ pain points and guide quantitative data collection.
Monitored online forums
Solicited community feedback
Built user empathy by playing the game
Reviewed CEO’s firsthand experience
Hung out at the water cooler
I monitored online discussions daily to capture user sentiment.
QUANTIAVE RESEARCH
Started with qualitative research to uncover users’ pain points and guid
Hypothesis
Shift and bulge
I monitored online discussions daily to capture user sentiment.
QUALTATIVE / QUANTATIVE PERSONAS
We divided the population into five macro groups.
Master / Grandmaster
Who: The top 5% of players, highly competitive and focused on skill mastery.
Findings: Primarily concerned with their exact rating; behave similarly to competitive chess players, valuing precision and accuracy above all.
Thoughts: Engagement for this group depends on features that reflect true skill; they interact only with elements that provide accurate, meaningful feedback on performance.
Platinum / Diamond
Who: Players below the top tier, on the right-hand slope of the skill distribution; often follow eSports and understand skill deeply.
Findings: Maintain a win rate above 50% and are highly engaged. Frustration from moving between tiers exists but contributes to the sense of accomplishment when progression is achieved. They enjoy displaying their medals and recognition.
Thoughts: This group represents the core audience. Design should reinforce progression milestones, highlight achievements, and support skill development to enhance the rewarding experience of advancing
Gold
Who: Players positioned just below the right-hand slope of the bell curve, predominantly Gold but occasionally approaching Platinum. They often perceive themselves near the top of Gold despite not yet reaching Platinum.
Findings: Maintain engagement with a moderate win rate around 50%. Enjoy displaying medals and recognition, and experience mild frustration as they fluctuate within Gold. This frustration contributes to a sense of accomplishment when progression is achieved.
Thoughts: Design should keep progression indicators intentionally vague to sustain motivation and positive self-perception, while still rewarding achievements within the Gold tier.
Silver
Who: Players on the left side of the bell curve; primarily Silver tier.
Findings: High drop-off occurs in Silver, with win rates below 50%. Progress feels slow, and the tier is often perceived as discouraging compared to Gold. We were surprised by the amount of churn, which was higher than expected.
Thoughts: This group experiences frustration and limited satisfaction. Design should aim to sustain engagement through small, achievable milestones and subtle feedback to maintain motivation.
Bronze
Who: Players primarily motivated by enjoyment and casual play; they appreciate perceivable progress.
Findings: Played more matches than expected. Wins typically reflected significant skill improvement rather than luck. Churn is less influenced by win rate and more by the lack of visible progression after victories.
Thoughts: Retention depends on highlighting incremental progress and skill development to sustain engagement and encourage continued play.
Players are slotted into seven different categories by rank. Each rank have ten sub ranks that they progress through before moving on to the next one.
DESIGN
Changes for Personas
Bronze / Silver: Replace ranking system with a progression system
Always provide one incremental award for winning, regardless how much skill their improve in the bell curve, and never take an award for losing.
Gold: Make it hard to fall back out of gold and easier to get into gold. Add a tournament system where they could earn a trophy of a higher rank.
Diamond / Platinum: Keep a pure ranking system and add a tournament system to give them a true esports feel.
Master / Grandmaster: No changes
Mini-progression
Every 3 months, players medals go now. they have to play a certain amount of games before they receive a medal and move up the ranks. This way, players have to play more games to get back where they were. Players liked this change because if feels like they are working their way up.
Matchmaking
Separate MMRs for each race per player. Each race, zerg, protoss and terran have very different play styles. Previously, players had one MMR regardless which race they place. The thinking was players skills were likely the same across races and should have the same MMR otherwise matchmaking wouldn’t feel balance and the overall matchmaking system would be hurt because there would be fewer data points for the system. The data scientists demonstrated how the matchmaking system was able to support the additional three different MMRs, so went that way.
MMRs and Leaving Games Early
Players would try to game the system to find ways to have easier matchers. Some of my solutions involved changing their matchmaking ratings to deter this behavior. The data scientists found that any one-sided or exhaserbated tweaks would damage the bell cuve.
As a result, I came up with one idea to not change MMRs if players leave a came within the first three minutes. The data scients said that would be okay. But, from a design perspective, that could potenitnailly benefit the higher ranked players, so I decided we would have this rule only apply to certain players. Togther, we found a sweet spot to find the best place. We wanted to deter them but not leave the system.
Reduce Leaving Games
To work around the not changing MMRs, we added a penalty so the user couldn’t play for a certain amount of time. Again we together to find a sweet spot to have this to implement this behavior but didn’t want to abritrairy attach number. We decided on a number and also added a rolling number that would increase repeat offenders. Again, we wanted to deter them but not leave the system.
RESULTS
Compare to the last major release, we saw substantial benefits across the board.
+40%
User retention
+500%
User engagement
88%
Metacritic score
PREVIOUS: Marriot-Starwood Merger