MMR 101

Intro to Matchmaking Rating

What is MMR?

Matchmaking Rating (MMR) is a number that represents player skill. Chess Elo (named after Arpad Elo; not an acronym) is a widely known example of an MMR system.

The expectations of an Elo-like MMR system are simple: players should gain MMR when they win, and lose MMR when they lose. The amount of MMR won or lost should depend on the relative rating of the opponent; e.g. defeating a higher-rated opponent rewards a player with more points than defeating a lower-rated opponent.

This article will cover the general principles and introductory math needed to determine how much MMR a player should gain or lose after each game, and will preview the main shortcomings of Elo-like rating systems with respect to modern online games.

Applications of MMR

Every PvP (player vs player) game needs a skill rating system for at least one, if not all, of the following cases:

1. Team-Balancing (TB)

Make the teams fair.

The importance of team-balancing is a design choice, but here are two reasons to consider it:

• Balanced matches statistically maximize each player’s effect on the game’s outcome, which typically increases engagement. Having a personally-relatively good or bad game matters more in a balanced match.
• Extremely lopsided matches reduce fun for the losers more than increase fun for the winners, and are more likely to drive churn than boost retention. (numerous studies; e.g. Kraaijenbrink 2009)

2. Bragging Rights

Tell players how good they are; reward skill.

Visible MMR is often called “Elo” by players, to mean it is a player-facing number based on skill rather than on time or money spent. Visible MMR is useful feedback for personal improvement, but a streamer’s professional credibility may also depend on it.

Skill feedback doesn’t always need to be a precise number though; it could be an icon, a tier, an interpolated superficial value based on MMR, etc.

3. Skill-Based Matchmaking (SBMM)

Put players of similar skill together.

The importance of SBMM increases when there is less room for team-balancing: with smaller team sizes or full-team parties. Or, where intra-team skill disparities or proficiency thresholds fundamentally change the way the game is played — advanced plays and team tactics emerge at higher skill levels.

SBMM can be a complex and contentious topic, but the root of the controversies mostly stem from the bragging rights issue — specifically, when SBMM is implemented without rewards or skill feedback, therefore punishing high skill players.

4. Analysis

Understand players better.

MMR is useful in all facets of gameplay analysis (weapon balancing, etc), as well as player segmentation in consumer-focused analytics.

In PvE, player ratings can also be used to assess bot and level difficulties.

MMR Requires Statistical Reasoning

Games are composed of complex, multi-dimensional interactions between players — and possibly random environmental factors. A simple number cannot perfectly capture or predict the results of these complex interactions.

Rock, Paper, Scissors

Consider rock-paper-scissors as played by three unimaginative players: Rock, Paper, and Scissors, who only choose their corresponding eponymous moves. What MMR values could represent their respective skills?

With only Rock and Paper, this would work:

Paper.mmr = 1
Rock.mmr = 0

Paper always wins, as predicted, with no uncertainty. But what about Scissors? To be perfectly predictive, Scissors.mmr would need to be simultaneously greater than 1 and less than 0.

With a one-dimensional number then, it is not possible to perfectly model and predict the outcome of Rock vs Paper vs Scissors. Even in a game this simple, the outcome is uncertain when MMR is the only information. (It is possible to perfectly model this example with two or three dimensional MMR, but that’s another topic)

Building a Performance Model

If each of Rock, Paper, and Scissors plays both of the other players the same number of times, all three will have identical win rates. It’s fair then to give all three players identical MMR values (say, zero).

Having identical MMR doesn’t mean the result will always be a tie (in three-player Rock vs Paper vs Scissors it will never be a tie) it means the outcome of a particular match is perfectly random and uncertain: a 50/50 toss-up from a pure MMR perspective.

This uncertain outcome can be modeled by assuming each player’s performance fluctuates per match. Sometimes they play well (say, +1, e.g. Rock vs Scissors), sometimes they play poorly (-1, Rock vs Paper), but average performance comes out to zero for all three players.

This performance model uses a discrete probability distribution, with a 50% chance of performing at +1, and a 50% chance of -1; therefore an expected value of zero.

Realistic Player Performance

Real player performance is more accurately modeled as a continuous distribution. If Player A and Player B have MMRs of 1.0 and 2.0 respectively, their performance probability distributions might look like this:

For each game, Player B’s performance is conceptually the x-value of a random point underneath the red curve, and Player A’s from under the blue. If B’s random performance is greater than A’s, then B wins. In this example, B clearly has a better chance of randomly picking a point with higher performance — and therefore winning — but exactly how much better remains to be seen.

Uncovering Player Skill

MMR is the system’s current guess about a player’s underlying skill.

Skill is a latent (hidden) variable. It’s not a directly measurable quantity; it’s a useful conceptual approximation. The only time “skill” can quantifiably be known is in simulation.

Ideally, with enough samples, MMR will converge on this hidden skill value and “uncover” it. The process to get there will be something like:

1. Predict the outcome of a match, given the MMR difference between players
2. Observe the actual outcome of the match (sample)
3. Correct each player’s MMR based on the difference between actual and predicted outcomes (update step)

And, repeat.

MMR Update (step three)

Working backwards from the correction step at the end, consider:

If all MMR values are perfectly accurate (equal to hidden player skill) then no correction should be required.

This simple condition is core to every MMR system.

To satisfy this equilibrium, the expected correction for winning must be canceled out by the expected correction for losing. To express it another way:

Where:

The probability of losing is the same as 1 minus the probability of winning:

So, the corrections for winning or losing must satisfy the equation:

Which can be solved generally as a linear function of the player’s probability of winning by introducing a parametric scale constant, k:

In plain English:

The reward for winning is proportional to the probability of losing.
The penalty for losing is proportional to the probability of winning.

Performance Observation (step two)

The correction amount can be simplified further, and expressed as a unified function of the outcome, if win/loss outcomes are expressed numerically:

The unified correction step is then:

In other words:

A player’s MMR update is linearly proportional to the difference between the actual outcome and the expected outcome (the probability of winning).

The value for k is tune-able, depending on the desired properties of the system: a large k value for rapid convergence at the expense of precision; a small k value for slow updates but eventual precision.

Steps two and three are ultimately straightforward: Observe the actual outcome (win, loss, or tie), and update the player’s MMR proportional to the difference between the actual outcome and the expected outcome.

The actual outcome is known directly, but how to get the expected outcome? This requires figuring out each player’s probability of winning as a function of their MMRs.

Probability of Winning (step one)

Going back to the performance model, what is the likelihood that Player B will outperform Player A?

MMR is a random variable, because it can have a different value every time it is sampled (looked at), which breaks the normal rules of arithmetic.

Random variables are the infinite hats that samples are pulled from. It’s not possible to simply add, subtract, or compare continuous random variables, because the underlying probability distribution has to be taken into account. Only the specific samples drawn from each hat in each round (each “experiment”) can be compared directly.

So, one way to answer the question about Player B’s probability of winning, P(A < B), would be to generate a lot of head-to-head experiments from A and B, and count how often the sample from B is greater than the sample from A. This is computationally expensive, and requires a random number generator and appropriate quantile function, but sometimes it’s the only option.

Some distributions though, like the normal distribution (aka “Gaussian”), have convenient properties that let them be compared analytically. The sum or difference of two normal distributions can be described as the sum or difference of the means, and the sum (always) of the variances:

Note: This is framed as “the probability of A less than B”, in order to integrate left-to-right.

In this new distribution, P(A < B) is equivalent to the area under the Probability Density Function (PDF) less than zero:

Which, is the integral of the PDF from -infinity to zero. This integral is also known as the Cumulative Distribution Function (CDF) of the distribution.

In this example, the means of A and B are 1.0 and 2.0 respectively, and the standard deviations (sigmas) are both 0.5. AB then has mean=-1 and sigma=0.7071.

The corresponding normal CDF(0) for AB is 0.92:

So, Player B has a 92% chance of winning against Player A, according to this MMR model.

Putting it all together: if Player B wins, they only win ~.08k MMR. If they lose, they lose ~.92k MMR.

Implementation Details

For reference, the full suite of functions for a normal distribution looks like:

The CDF (Cumulative Distribution Function) is the only function necessary for the performance model in the preceding example, but it can be difficult to implement — depending on the environment and math libraries available — due to its dependence on the normal error function (erf), which is an integral that cannot be calculated analytically. There are several acceptable options for approximating erf though (outside the scope of this article).

A logistic distribution is another common choice for a performance model, although the distributions can’t be added or subtracted as simply. The logistic CDF is much easier to implement, especially for environments with a built-in tanh function:

Performance is not Enough (the problem with Elo)

A performance model may seem like the only component needed to build a basic MMR system… and it is, under specific conditions: e.g. a 1v1 game where all players play against a perfectly random sample of other players who don’t improve over time.

However, as SBMM (or team-balancing) approaches perfection, the performance model becomes completely irrelevant.

For example, Elo might be calibrated so that a difference of 400 MMR units equates to a 90% win rate — that gives MMR a relative meaning. If SBMM or team-balancing is perfect though, then the actual MMR differences are always 0, and the expected win rates are always 50%. The correction step for winning or losing is therefore always +/- 0.5k, regardless of how the performance model is calibrated. The performance model never comes into play, so it has absolutely no influence over the ultimate distribution of MMR.

As this system runs, the population distribution of MMR will expand unpredictably. This is obviously unacceptable in a competitive online game, where MMR is expected to correspond to fixed MMR tiers (bronze, silver, gold, etc).

For Elo (or TrueSkill) to not degenerate, matchmaking needs to have a stable amount of randomness (it can’t be perfect), or the MMR population needs to be sufficiently initialized to provide reference points for new players to compare against. New players also need to come into the system at a relative trickle — a large influx of new players, like at the start of a ranked season, will disrupt the system unpredictably.

To Be Continued…

SBMM isn’t the only hurdle for an MMR system. Asymmetric player improvement also leads to inevitable skill drift and inflation in Elo-like systems (due to relying on relative MMRs).

The solution to all of these problems is to introduce a population model that works in concert with the performance model. A robust system needs to have both.

Follow-up articles will expound on population models, simulation development, analysis of real player data, and alternative MMR systems.