in orbit

I mostly talk about video games and the world wide web

Mar62009

Game Publisher Consistency

Dave Perry recently posted some game review scoring stats gleaned by an independent source in his blog. The list of 20 publisher (the full, unreleased list contains many more) shows the top and bottom 10 publishers based on their games' Metacritic scores which are of course a weighted average of review scores from various sites.

The results are interesting for a variety of reasons. First off the top score (held by Rockstar) is 19. The system works by awarding positive, negative or zero points based on review scores. In R*'s case they have 23 titles in Metacritic. Six of them are worth 2 points, ten worth 1 point each and three of those points are canceled out by three D-rated games (worth -1). If you're wondering why that only adds up to 19 games it's because four of them simply don't count for or against the score. This seems off to me, but we'll just run with it for now.

Regardless, R* has the top score clocking in at a whopping 19 points. The bottom score is held by Ubisoft with an impressive(ly depressing?) -148 points. For those of you keeping track that puts the median score around -65, which means that the vast majority of video games are shit.

My second issue with the data is how it doesn't quite seem 'right'. For instance look at the #3 and #4 ranked publishers. Blizzard, highly commended for their addictive, polished and downright beautiful games, has a score of 11. Microsoft comes very close to Bilzzard with 10 points. However looking only slightly deeper we see the apparent flaws in this system. Blizzard's 11 points come from a meager sevem titles, four for 2 points, three for 1 point each. Blizzard has zero games that didn't count or counted for negative points, which would seem to mean they put out only a few games (less than one per year) but those games are of very high quality. Conversely Microsoft's 10 points comes from a set of 110 games. They published 103 more games than Blizzard since 2000 but also earned one fewer point. So it would seem the distribution of points among Microsoft's games is nowhere near as good as Blizzard's. Indeed if you look at the breakdown the majority of MS published games fall in B-range with the rest mostly distributed below B and only eight (7% of total) getting an A score.

A perhaps more accurate gauge of how consistent a publisher is would be to look at the average score per game, and indeed this paints a much different picture of the stats:

PublisherAvg. Score
2D Boy2.00
Blizzard1.57
Stardock1.17
Rockstar0.82
Valve0.75
Telltale0.61
Bethesda0.28
Square-Enix0.11
Microsoft0.09
Nintendo0.06

This places Microsoft much lower than Blizzard by quite a bit. In fact MS's per-game average is below 0.1, as is Nintendo's (which is even lower). This is more or less intuitive based on the number of titles each publisher puts their name on. There are a bunch of AAA titles (Gears of War, Halo etc for MS, Mario Galaxy, Twilight Princess et al for Nintendo) that bring up the overall score enough that the publisher remains consistent in the eyes of the calculation.

Keep in mind that the maximum possible average (obtained by 2D Boy) is 2.0, so the fact that Blizzard is hovering near 1.6 is very impressive. Speaking of 2D Boy, it's also worth noting that they should probably be dropped from the list entirely because their publishing efforts consist of two titles; the PC and Wii versions of World of Goo. Both of those scored very well but with such a small sample consisting of what is realistically only a single game doesn't seem to be statisically relevant.

And so, if you were looking for the most consistent publisher in video games that would be Blizzard, as one might expect. However I'd hesitate to suggest these stats, or even Metacritic in general would be a good gauge of much of anything. Essentially throwing away an entire set of scores for each publisher (the C-level games) as well as not weighting anything in any significant way based on the volume of games makes it suspect. However I'm sure there is some useful info to be gleaned from Metacritic, though I'm not going to hold my breath thanks to arbitrary score scales or bizarre scoring systems that many game review sites use. All that we cab really gather from this data is that there are a lot of games out there and the majority of them are not very good, and I confess I already knew that before we started. Video games!
Add Comment
Name:

URL (optional):

Your Comment: