Is PS4 actually "easier"? Statistical analysis of Q School
Oct 28, 2018 2:00:11 GMT -5
mes5107, Driver8, and 9 more like this
Post by Deleted on Oct 28, 2018 2:00:11 GMT -5
Just spent over 2 hours putting this together, but this global Q school is probably the best chance we'll ever get of comparing platforms in a fair way so it's worth it!
Lots of people have claimed that PS4 is "easier". Others have said any score differences we see from eyeballing the top of leaderboards are purely down to the fact that there's a larger player-base on PS4. Fortunately there's a very straightforward statistical method to test this latter assumption, which implies that, if the population sizes were the same, the score distributions would be roughly equal (within a certain measurable margin of error). The T-Test below concludes, statistically significant to the 0.0001 level (which is as significant as any test needs to be), that overall PS4 scores are better than the other 2 platforms. Note I am not saying it's easier, because 'easy' is a loaded term that can be abused in a gaming context. I am only saying that we have extremely strong evidence that overall, PS4 scores are better even when controlling for how many players are on each platform. I presume nobody will claim with a straight face that there are fundamental differences in the ability of moving a stick up and down between people who buy PS4s versus those who buy other gaming platforms
Some vital caveats before I show the data:
1. You cannot make any conclusions about individuals from aggregate statistical data. This is such an important point, and if people absorbed it, millions of hours of pointless arguments around the world would be saved. I'm not saying it's 'stupid' or 'morally wrong' to make such conclusions (although it can be depending on context), I am saying it's literally a logical impossibility. Aggregate statistics are abstract, over-simplified representations, they are not real, empirically identifiable 'things' in the world. The best summary of this point about scientific or statistical models comes from George Box who famously said "All models are wrong, but some are useful".
So let's say someone claims that overall, Dutch people are taller than Indonesian people. The statistics support such a claim for the overall population, but they can't tell us anything about individuals - you could have a 7ft Indonesian or a 4ft Dutch person within that population. That doesn't make the statistics incorrect per se, because the statistics are simply representations, they are not the people themselves. Conversely, an employer at an exclusive golf club may look at a job applicant with a working class background and cite statistics that in general, working class employees are 20% more likely to quit the role within a few months, speculating that perhaps this is because they struggle to fit into the culture. But that would be an unreasonable approach to making a hiring decision, because that particular individual may have significant experience and aptitude for building rapport with all manner of people, including middle-upper class golfers. The statistic can provide an interesting or useful overview of a very large and complex population of people, but it cannot tell you anything about an individual, and it certainly shouldn't inform specific decisions or conclusions about specific people.
2. I used data only for master club users, because pro and beginner clubs distort the data making any comparisons unfair.
3. This data only tells us the 'what', and not the 'why'. So no you can't say that this proves any platform is "easier", only that scores are better on that platform even when controlling for the number of players. Anecdotal reports may suggest that tempo is slightly easier on average with a DS4 controller. EDIT: Jeff, TGCT data overlord, has kindly provided stats in post 13 below for tempo across platforms, which shows that PS4 players do get perfect downswing tempos more often than PC and XB1 players.
Anyway, that said, here's a box plot summary chart showing the distribution of scores, firstly for all players then for each individual platform. In brief, the box area is where most scores tend to be, the line edges are 1st-4th quartiles, the 'x' is the raw mean score, and the line within the box is the median. The individual dots are for outliers - you'll see that even though most PS4 players score slightly lower than other platforms, across the whole PS4 population they have a lot more variance particularly at the higher end of scoring, so this just underscores the point that getting a PS4 alone does not guarantee that you will score better. Statistics aren't individuals.
The mean and stdev statistics for each platform:
PS4: Average 71.64 with a standard deviation of 5.44
PC: Average 73.17 with a standard deviation of 5.73
XB1: Average 73.89 with a standard deviation of 4.98
Just from eyeballing the above we can see that there's no difference worth investigating between PC and XB1 scores, because their standard deviations overlap their mean scores. Not the case with PS4 however. So I combined PC and XB1 scores (N=595) and conducted a T-test comparison, which controls for population sizes, against PS4 scores (N=783), resulting in a T score of 6.84. The probability of this result, assuming the null hypothesis ("there is no difference in scoring between platforms, assuming even player-base size"), is less than .0001. Another way of putting it is that we can be 99.999% confident that there is a real difference between PS4 versus the other 2 platforms, which cannot be attributed to random chance. It's not a big difference, and we can't make any conclusions as to why this might be the case from this data alone, but at the very least we can be extremely certain that the (slightly) better PS4 scores are not simply due to the fact that there are more PS4 players.
Anyway, hope this helps resolve at least one aspect of this debate, and I hope that all the cautious caveats I've included in this post will prevent any unnecessary paranoia, conspiracy theorising, unreasonable claims or attacks against people using other platforms. Wait, who am I kidding, this is the TGCT forum...
Lots of people have claimed that PS4 is "easier". Others have said any score differences we see from eyeballing the top of leaderboards are purely down to the fact that there's a larger player-base on PS4. Fortunately there's a very straightforward statistical method to test this latter assumption, which implies that, if the population sizes were the same, the score distributions would be roughly equal (within a certain measurable margin of error). The T-Test below concludes, statistically significant to the 0.0001 level (which is as significant as any test needs to be), that overall PS4 scores are better than the other 2 platforms. Note I am not saying it's easier, because 'easy' is a loaded term that can be abused in a gaming context. I am only saying that we have extremely strong evidence that overall, PS4 scores are better even when controlling for how many players are on each platform. I presume nobody will claim with a straight face that there are fundamental differences in the ability of moving a stick up and down between people who buy PS4s versus those who buy other gaming platforms
Some vital caveats before I show the data:
1. You cannot make any conclusions about individuals from aggregate statistical data. This is such an important point, and if people absorbed it, millions of hours of pointless arguments around the world would be saved. I'm not saying it's 'stupid' or 'morally wrong' to make such conclusions (although it can be depending on context), I am saying it's literally a logical impossibility. Aggregate statistics are abstract, over-simplified representations, they are not real, empirically identifiable 'things' in the world. The best summary of this point about scientific or statistical models comes from George Box who famously said "All models are wrong, but some are useful".
So let's say someone claims that overall, Dutch people are taller than Indonesian people. The statistics support such a claim for the overall population, but they can't tell us anything about individuals - you could have a 7ft Indonesian or a 4ft Dutch person within that population. That doesn't make the statistics incorrect per se, because the statistics are simply representations, they are not the people themselves. Conversely, an employer at an exclusive golf club may look at a job applicant with a working class background and cite statistics that in general, working class employees are 20% more likely to quit the role within a few months, speculating that perhaps this is because they struggle to fit into the culture. But that would be an unreasonable approach to making a hiring decision, because that particular individual may have significant experience and aptitude for building rapport with all manner of people, including middle-upper class golfers. The statistic can provide an interesting or useful overview of a very large and complex population of people, but it cannot tell you anything about an individual, and it certainly shouldn't inform specific decisions or conclusions about specific people.
2. I used data only for master club users, because pro and beginner clubs distort the data making any comparisons unfair.
3. This data only tells us the 'what', and not the 'why'. So no you can't say that this proves any platform is "easier", only that scores are better on that platform even when controlling for the number of players. Anecdotal reports may suggest that tempo is slightly easier on average with a DS4 controller. EDIT: Jeff, TGCT data overlord, has kindly provided stats in post 13 below for tempo across platforms, which shows that PS4 players do get perfect downswing tempos more often than PC and XB1 players.
Anyway, that said, here's a box plot summary chart showing the distribution of scores, firstly for all players then for each individual platform. In brief, the box area is where most scores tend to be, the line edges are 1st-4th quartiles, the 'x' is the raw mean score, and the line within the box is the median. The individual dots are for outliers - you'll see that even though most PS4 players score slightly lower than other platforms, across the whole PS4 population they have a lot more variance particularly at the higher end of scoring, so this just underscores the point that getting a PS4 alone does not guarantee that you will score better. Statistics aren't individuals.
The mean and stdev statistics for each platform:
PS4: Average 71.64 with a standard deviation of 5.44
PC: Average 73.17 with a standard deviation of 5.73
XB1: Average 73.89 with a standard deviation of 4.98
Just from eyeballing the above we can see that there's no difference worth investigating between PC and XB1 scores, because their standard deviations overlap their mean scores. Not the case with PS4 however. So I combined PC and XB1 scores (N=595) and conducted a T-test comparison, which controls for population sizes, against PS4 scores (N=783), resulting in a T score of 6.84. The probability of this result, assuming the null hypothesis ("there is no difference in scoring between platforms, assuming even player-base size"), is less than .0001. Another way of putting it is that we can be 99.999% confident that there is a real difference between PS4 versus the other 2 platforms, which cannot be attributed to random chance. It's not a big difference, and we can't make any conclusions as to why this might be the case from this data alone, but at the very least we can be extremely certain that the (slightly) better PS4 scores are not simply due to the fact that there are more PS4 players.
Anyway, hope this helps resolve at least one aspect of this debate, and I hope that all the cautious caveats I've included in this post will prevent any unnecessary paranoia, conspiracy theorising, unreasonable claims or attacks against people using other platforms. Wait, who am I kidding, this is the TGCT forum...