Imbalance and Uncertainty

 Hello, dear readers. I had meant to post a 2nd time last week, but fell behind on that. I apologize to all 0 of you for that.

Yesterday I ran across a graphic that touched on the balance of current Games Workshop games (Just AoS and 40k), and since that's what had been on my mind I wanted to share my immediate thoughts and reactions to the current state of things, as it were.

THE STATE OF THINGS

As it were

from what data we have

that I personally haven't actually analyzed

and looking at it through entirely biased and subjective lenses

40k win rates by faction

Age of Sigmar win rates by faction

So, for those of you who remember from last week, Age of Sigmar's balance was considered bad enough that GW released a points incentive for playing weaker armies and against playing powerful models. I was honestly impressed, however, at how balanced Age of Sigmar appears to be, at least compared to 40k. The first graphic there makes some assumptions about critical balance problems, and a gives what they consider an acceptable range - all armies having win rates between 45 and 55%. There is, unfortunately or fortunately depending on what you enjoy, a lot more going on.

I'm not particularly naïve when it comes to statistics. My undergraduate degree was math heavy, statistics specifically, and my graduate work and most recent professional work has all been data analysis intensive. I say that to let you know I am not simply talking out of my ass when I say the data I've been able to find is far from sufficient to say how good or bad different factions are. The last time I played around with a comprehensive dataset was looking at WTC stats for Warmachine, which included player - faction - caster variables for every game. This is a two day event with hundreds of players, so it isn't exactly small. The problem with this kind of data is that these three things are, by virtue of how wargames work, entirely correlated.

What does that mean? Essentially, that if a player is playing X caster, you know they're playing Faction A. If you're looking at Player B, they're always playing Faction Y, and either Caster B or Caster C. This high correlation between variables means it is incredibly difficult to parse out the effects of any one. Is it Player A, Faction Y, or Caster C that is influencing the odds of winning, and by how much?

The answer, of course, is Yes. If Player A is a competitively minded player, and Caster C is particularly powerful into the current play environment, and Caster C is in Faction Y, then you're likely to see all three together. Caster + Faction pairs were, at the time, entirely a given. Each caster was in one faction, so you didn't see them in different Caster | Faction pairs. Further, at a single event like this, players are entirely linked to their one faction, and two casters. 

Enter, modeling

Naive Bayes is one of the quicker ways to create a classifier from such a set of data. In this case, you use it to look at your features/variables (Player1, caster, faction, player 2, caster faction) and based on a learning/larger data set, you place the match into a "class". In the case of a game, this will be a win/loss - so the classifier is a way to take the pieces of a matchup and, given some assumptions, try to predict the outcome. 

After playing around with it using different learning sets to build the classifiers, looking at different variables and variable combinations, the strongest model I could find to predict the outcome of a match was just Player Vs Player. Which, honestly is kind of garbage. Players who won more are more likely to win than players who won less. I wish I had been able to say more than that, but model selection criteria really didn't say much about other models (see: Information Criterion if you're interested).


So! What does this mean? Certainly if you looked at the data you'd see some standout factions and standout casters, if looking solely at win %s for those casters. But the reality is way more complicated. There are, of course, even more things you could bring into it, and team events are a different animal altogether. That said, my point here is that correlation between all of the variables that go into a matchup are massive. I did look at more ways to analyze the data, one of the more popular visualizations in this sort of many-variable problem is what's called a PCA. PCA essentially condenses a multi-dimensional space into a 2D one, using a bunch of fancy math involving eigenvectors. Each dimension is a variable, and when you condense it down you're combining portions of each one to get a "Principal Component", that is, a combination of the correlation of these different variables that gives you the strongest "axis".

Wherein I go into too much detail on PCA

Or not enough

this is a very vague explanation of it

Data in 3-dimensional space

2D plane through the 3D space

2D graph of the data

In actual practice you usually have way more variables/dimensions, and the first two principal components might explain a few % of the spread (more likely when the variables are all truly independent). In the case of this example, though, you can see that the dots are heavily spread out along the X (green), and less heavily along the Y (Blue line here). Using this graph, you can come up with some rules for predicting if a dot will be red or blue based on it's X and Y values, and then test it on datasets to see how accurate you are. 

Gut Reaction

When I first saw that graph, I was honestly left with a gross feeling. I was wondering if it was worth even continuing with my 40k hobby projects - I'm currently printing an army that I sculpted. If the game is going to continue to be a mess of imbalance and horribly skewed matchups, do I really want to bother with it? Or even worse, if I end up on the high end of the balance curve do I want to feel like a dick for playing an army I wanted to play - and am I okay with doing this to other people?

Balance is hard, and balance in 40k is clearly skewed - Top players don't choose their armies arbitrarily. But the fact that all of these variables are so intertwined makes things look significantly worse than they actually are. Are Harlequins really so amazing that they can pull an 80% win rate? Maybe, but people who take the game really seriously and perhaps even people who got early looks at the books/rules, and managed to grind a bunch of games quickly.... thought it looked strong, and moved over to it. If everyone just had a randomly assigned faction, it seems unlikely they'd be winning 80% of matches. It's an unfortunate fact that minor balance issues are exacerbated by players who are able to exploit even the smallest advantages. 

Well, dear reader, there's my rambling screed on this topic. It's been some years since I did any serious statistics, I've literally died in the interim and forgot numbers for awhile afterward. Some of this stuff might be oversimplified, or outright wrong. If you're a smart person who wants to correct me, please leave a comment doing so. I'd appreciate it. And if you're worried about balance, try not to worry so much. If it becomes a problem in your play group, I sympathize, you can try talking about it with them or if that doesn't work then... well honestly one of the things I think is underutilized is just how many games are out there. Find something that you enjoy, don't feel stuck playing something that makes you unhappy just because of inertia and familiarity.