Data-Driven design

Directed data collection to generate insights that informed strategic design decisions across multiple projects and touchpoints.
Role: Lead UI /UX Designer, research and define critical user pain points, guiding a team of six to analyze data and shape design direction

Goal

To deliver a major content release that addressed churn, balanced competitive and casual engagement, and strengthened the overall player experience. Increase engagement and reduce the rate of decay from previous releases

CRITICAL CONTEXT

Market-Pioneering Discovery in 2010

In 2010, competitive modeling in games relied on simple models that assumed a static, normal user distribution. While analyzing proprietary data, our consulting Doctor of Mathematics, the first ever hired by a game company, was possibly the first to quantify, measure, and model the emerging 'shift and bulge' in user data, a phenomenon that is common knowledge today but not yet known or not yet discussed prior to 2010.

Invitation to Wharton School of Business

I was invited to the Wharton School of Business by Kevin Werbach, author of “For the Win”, to discuss this topic among others regarding game thinking in the business world, a concept known as gamification.

Problem Statements

Competitive Persona

In the competitive part of the system, lower-skilled players were leaving at a higher rate than the rest of the population. While many assumed that lower-skilled players leaving was inevitable, we treated it as a real problem that could be addressed through thoughtful, data-informed design.

Social Persona

Engagement was low, and early churn was high among the casual and social player base.

The Team

Our project was driven by a cross-functional team, including:

  • Doctor of Mathematics, Ph.D.: We were the first game company to have hired a doctor of mathematics. He applied deep expertise in probability and curve functions.

  • Data Scientists (2): Focused on data pipeline to collect data for analysis.

  • Developers (3): Responsible for implementing the required instrumentation for data tracking and deploying the final design features.

  • Myself: Conducted the initial qualitative research, formulated the core hypotheses, led collaborative analysis to interpret the quantitative data, and drove the final design and actionable solutions.

DESIGN PROCESS

Research

Hypothesize

Design

Test

USER RESEARCH

I began with qualitative research to understand users’ pain points and determine which numerical data to collect, guiding the design process.

  • Monitor online forums

  • Solicit community feedback

  • Build user empathy

  • Review CEO’s firsthand experience

  • Stay around the water cooler

I monitored online discussions daily to capture user sentiment.

PERSONAS

Reverse Engineers

Who: Two people, one from MIT and the other from UC Berkeley who like to keep track of any changes we would make to the system. They would write on board forums and write on their blog.

Findings: They were very accurate in their findings.

Thoughts: They would have a firm understanding of what we were doing. Even if we wanted to prevent them, we couldn’t.

Elo

Who: People who want to know what the Reverse Engineers say. They are like chess players in that they only care about their number and not the rank we assign them.

Findings: They were the top people at the very right side of the bell curve (typically borderline Master, Master, Grandmaster). Some would be professional eSports players that want play anoynmaly for fun. but there also a lot of those the middle of the bell curve that cared. They care about their elo and the top.

Thoughts: We wanted to give these guys as accurate system as possible. The top of the user base was easy to support and we could do a good job. The middle people cared but they didn’t know how the system worked and were fine they number we gave them even if it was wrong.

Medalist

Who: People who don’t know about elo, trust us, and care about the rank we assign them (Bronze, Silver, Gold, Platinum Diamond)

Findings: This was the majority of the audience. They cared about their ranking and brag about it. Quite a lot would say I’m in [Gold] but I’m really top [Gold]. When we checked the numbers, they were nowhere near the top but they still had fun, for the most part but we still they could be in danger of churning.

Thoughts: Our core audience was here. They were having fun but we didn’t want them to feel like they were stuck and stop playing because they feel stuf. We think we could do better with them but weren’t too worried because they had fairly close to a 50% win ratio. It was mostly about how much fun with the game and how far they started slipping from the 50%.

Uber-Casual

Who: This is where our CEO fell in. People who just liked the game and didn’t seem to care about losing and just had fun playing.

Findings: The uber-casual would play quite a few games and we’d see their skill increase drastically. the equivalent of going from going from gold to platinum, but they were still in bronze. This is common knowledge in the sense it was very casual common knowledge where people wouldn’t give it a second thought.

Thoughts: These guys were increasing at an huge rate but we didn’t give them the medal to represent their massive skill gain. Also, they don’t know about elo or care about it.

HYPOTHESIS

Medalist: Give them more positive feedback and rewards. Give them reasons to brag, post on social media about their excitement. These guys will stick around if the Elo people are happy and they’re getting medals.

Uber Casual: If we gave them positive reinforcement, they would play more games. It didn’t have to come to win ratio. If they got positive reinforcement, they would still be inclined to play. When they do, the system will be more healthy overall.

AMAZING DISCOVERY

1. The shift

After collecting the data and taking a quick a look at it, the analysts realized the bell curve was shifting to the right. To our knowledge, no one had every seen this or at least they had never talked about it. No other game had the number of players our game has and no one has the team do the analysis we could do.

2. The bulge

Aside from the shift, the normal distribution wasn’t normal anymore. The top peak was followed by a trough, a smaller peak, followed by the rest of the normal distruption. Again, this is not someone has evet talked about.

The Implication

adf

The solutions

  1. We reset the curve, but we knew we could reset it daily. We decided 3-4 months would be decent spot, so we decided 3 months

  2. Designers could use that reset to our benefit.

DATA DISCOVERY

  • Lack of rewards

  • Lack of sense of progress

What we

When the bell curve shifts, the left-most curve does not shift 1:1; rather, the tail shrinks and contracts as lower-skilled players leave the system. This structural decay—a major focus of competitive game design analysis in the early 2010s following the launch of titles like StarCraft II (2010)—drives a self-accelerating cycle of player churn and skill inflation.

Churn in the bottom 3% was high possibly due to lack of positive reinforcements, low win rate, lack of interest in competitive play, or other unknown reasons.

DESIGN SOLUTION

  • Shift

  • Bulge

  • Lack of rewards

  • Lack of sense of progress

When the bell curve shifts, the left-most curve does not shift 1:1; rather, the tail shrinks and contracts as lower-skilled players leave the system. This structural decay—a major focus of competitive game design analysis in the early 2010s following the launch of titles like StarCraft II (2010)—drives a self-accelerating cycle of player churn and skill inflation.

Churn in the bottom 3% was high possibly due to lack of positive reinforcements, low win rate, lack of interest in competitive play, or other unknown reasons.

CHANGE HIGHLIGHTS

  • Map personas in real-time to deliver targeted, custom experiences

  • Developed extensive new content to engage a wider audience

  • Improved and future-proofed the matchmaking system

  • Created new UI to guide and incentivize players

  • Expanded on the most popular rewards and introduced complementary ones

  • Reworked social features to maximize and maintain interactions

When the bell curve shifts, the left-most curve does not shift 1:1; rather, the tail shrinks and contracts as lower-skilled players leave the system. This structural decay—a major focus of competitive game design analysis in the early 2010s following the launch of titles like StarCraft II (2010)—drives a self-accelerating cycle of player churn and skill inflation.

Churn in the bottom 3% was high possibly due to lack of positive reinforcements, low win rate, lack of interest in competitive play, or other unknown reasons.

RESULTS

  • Map personas in real-time to deliver targeted, custom experiences

  • Developed extensive new content to engage a wider audience

  • Improved and future-proofed the matchmaking system

  • Created new UI to guide and incentivize players

  • Expanded on the most popular rewards and introduced complementary ones

  • Reworked social features to maximize and maintain interactions

When the bell curve shifts, the left-most curve does not shift 1:1; rather, the tail shrinks and contracts as lower-skilled players leave the system. This structural decay—a major focus of competitive game design analysis in the early 2010s following the launch of titles like StarCraft II (2010)—drives a self-accelerating cycle of player churn and skill inflation.

Churn in the bottom 3% was high possibly due to lack of positive reinforcements, low win rate, lack of interest in competitive play, or other unknown reasons.

CHANGE HIGHLIGHTS

  • When a competitive ranking system's bell curve shifts, the left-most curve, representing the lowest-skilled players, does not shift 1:1; instead, the tail contracts and depopulates.

  • This structural decay, a critical subject of developer analysis and community dissection, is strongly linked to the launch and maturation of large-scale competitive titles like StarCraft II (2010), which popularized the modern tiered ladder system. The vast player data and public scrutiny of StarCraft II's ranking system provided the foundational evidence for why new players churned and how the entire skill distribution became artificially inflated—a process where lower-skilled players leave, the average skill of the remaining pool rises (shifting the curve right), and a "bulge" of veteran players forms at mid-to-high ranks.

When the bell curve shifts, the left-most curve does not shift 1:1; rather, the tail shrinks and contracts as lower-skilled players leave the system. This structural decay—a major focus of competitive game design analysis in the early 2010s following the launch of titles like StarCraft II (2010)—drives a self-accelerating cycle of player churn and skill inflation.

When a competitive ranking system's bell curve shifts, the left-most curve does not shift 1:1; instead, the tail contracts and depopulates. This structural decay—where the overall skill distribution inflates and the lower end is starved of players—was a critical subject of analysis that emerged in the early 2010s, directly following the launch of large-scale competitive titles like StarCraft II (2010). Crucially, the deepest initial understanding of this issue, including the identification of the recursive churn and right-side "bulge", came from teams who applied rigorous, high-level mathematical expertise (such as a doctor of mathematics) to dissecting the live player data, giving your community a lead on diagnosing this fundamental competitive flaw before it was widely understood by the rest of the industry.

Churn in the bottom 3% was high possibly due to lack of positive reinforcements, low win rate, lack of interest in competitive play, or other unknown reasons.

CASE STUDY

SOCIAL PLAY

Ranked play was a key staple of the product. Competitive gaming in the first installment was so popular that it helped spark today’s $6B+ eSports industry.

ANALYSIS

  • USER RESEARCH: Lower-skilled players found the ranking system frustrating.

  • HYPOTHESIS: The system experienced recursive churn as lower-end users along the normal distribution stopped playing.

  • DATA: The lowest 5% saw little reward for wins despite skill gains, resulting in attrition.

  • DATA: Over time, the normal distribution shifted in a way that disadvantaged the lower half in terms of rewards.

SOLUTIONS

  • Provide more positive reinforcement for the bottom 25% of players.

  • Automatically readjust the normal distribution every three months to maintain the desired experience and future-proof the system.

  • Hired a new game designer to create a low-pressure content, encouraging players to stay engaged in the game and possibly return to competitive play.

8% of the users received a disproportionately high share of development effort.

This YouTube video highlights changes and reasonings behind them. Later, I gave a similar presentation at the Wharton School of Business on gamification.

Added rewards for eSports-focused personas.

Launched a major casual content initiative with a new designer on the team.

RECEPTION

YouTube comments show user excitement over some upcoming UI changes over social features and content for casual personas.

RESULTS

+40%

User retention

+600%

User engagement

88%

Metacritic score