INSIDE MATCHMAKING
By Cat Varner on April 22, 2016 Measuring Skill: Part I
Skill-based matchmaking is a buzz term that can earn you a lot of impressed looks if you weave it into the conversation at a party. How do you actually measure skill in a video game though? As a statistician and a long-time MOBA gamer, I was looking forward to building a model with tons of factors that I thought could predict skill in this game genre.
The Goals
We want to make fair matches, but we also don't want to make players wait forever for a match. We need to adjust new players to their true skill relatively quickly, but we also want the system to respond to changes in player skill over time. We want to create a competitive environment where players can try hard and feel a sense of skill progression in our future ranked draft mode. How it Works
We realized not everything that might predict skill can be used in a matchmaking system. It is important that anything we include cannot be easily exploited so that players won't change how they play if they know how the rating algorithm works. For example, we don't want players to prolong their games just to pad their KDAs or destroy every tower before attacking the core.
Our current model does the following
Averages individual ratings across team. Determines adjustment values based on expected probability of winning. Adjusts faster during initial placement matches (variable k-factor). Includes social party adjustment. Expands rating requirements based on time spent in queue. Your matchmaking rating (MMR) is affected by whether you win or lose against opponents of similar skill. Your individual rating is averaged with your teammates' ratings to create a team score, although we are testing other aggregation techniques as well. Similar to an Elo System, you gain more points if you win against a team with a higher rating. Your rating moves the fastest when you are playing your first few PvP matches, as we are learning the most about your skill early on.
During Online Tests, we saw a lot of players who were teamed up in large social parties winning more often than players of the same skill who were riding solo. we responded by adding a social party adjustment that finds you harder opponents when you're in larger and larger social parties since you're likely coordinating with each other. We're keeping an eye on the win rates of different social party sizes to make sure that the party factor stays balanced.
The last part of the current matchmaking system is to make sure it can perform well at fluctuating queue volumes. Since we can't expect everyone to be playing Paragon all of the time (Do it!), we need to make sure that our system expands its MMR requirements at a decent pace to eventually prioritize making matches at all once players have been waiting for awhile. We have found that highly competitive teams would rather wait longer in order to find a fair matchup than the average player, so we recently worked on a formula that will weight finding fair matches stronger at higher MMRs (Coming to you live soon™!).
The Data
We use a lot of metrics to gauge the health of matchmaking, but here are a few of them for you fellow math nerds out there. First, we want to see whether our ratings are predictive, as in how often does the team that we think has a higher rating actually end up winning? We calculate an expected score for each game that represents the odds that one team will win based on the difference in the two teams' MMRs (e.g. a team with an expected score of 60 should be winning around 60% of the time).
If our system was exactly perfect at predicting skill, the trendline of expected score to actual win rate would have a slope of 1. Early Access is 1.2 so far. Our Early Access trendline has been a general improvement over the Online Tests. Our model performs better when predicting fairly close matches. Another main matchmaking metric is how often we are able to make fair matches. Depending on how many players happen to be in the queue at the same time, what Heroes they're playing, who they're partied with, etc., we may not always be able to find 50/50 fairness but we try for at least 45/55 for now.
In Early Access, we have found 45/55 or better matches 81% of the time. One of our main goals with matchmaking is to keep improving this percentage. Lastly, matchmaking times are something we always watch. On our latest build, our median wait time has been 1.5 minutes, with 75 percent of matches being made within 3 minutes. The longest 5 percent of matches are taking more than 6.5 minutes. We want to improve our messaging of your estimated time in queue, but we'd like to prioritize finding you fairer matches before we start trying to lower queue times by too much.
Next Steps
We are currently in the throes of designing our ranked draft mode. It is one of the most important things we can do to make sure that Paragon is a competitive, long-lasting MOBA game. As Paragon continues to evolve, measuring player skill will need to evolve alongside it.
One of the changes coming up soon is having team composition checks in the teammate search phase of matchmaking. There will be checks in place to make sure your team has at least one tank, one ranger, etc. This should help you avoid making you play the classic 5 melee comp, unless a party of friends purposely goes into the queue like that.
Our current system has a technical constraint that each team can only have one social party, so every duo queue has to be filled with only solo players. This increases our matchmaking times and makes it harder to find fair matches because high rating players tend to only play in social parties. Investing significant development time into removing this constraint should do a lot to improve the fairness of matches.
Lastly, thank you to everyone in the Paragon community for your feedback on our matchmaking system! As a work-in-progress, the system's greatest strength is how much data we're collecting from our players every day. We want to continually improve the matchmaking system, but we want to make sure we do right by you in the process. I'll make sure to keep you guys updated with our progress along the way and interesting things we find, so stay tuned!