Hey, CreaturesOfComfort:
Appreciate the feedback. Truly, it allows for constructive dialogue. Before I respond to this, I'm curious as to what your academic and research background is in. I presume statistics since this's what you pointed out? What year are you? Have you conducted any research that you'd like to share (always nice to learn something new)? Do you have an opinion on the creation of a constructive plan on how to work around inherit self-selection issues tied to online research? I'm genuinely curious--let me know.
One minor correction to your statement about myself. I was trained in Psychology (got my B.S. as an undergrad, minored in Philosophy), getting a M.S. in Public Communication & Technology, and have worked in psychology, communication, anthropology, sociology, philosophy, etc. I'm rather interdisciplinary and have indeed taken my fair share of research methods and statistics courses. Though that's neither here nor there. Learning some Anthro stuff but am not an Anthropologist currently.
But I digress.
The thing about surveying online samples is that is difficult to get a sample that is truly randomized. This's why non-probability based convincing snowball type sampling has to be utilized. It's nearly impossible to obtain true probability sampling with online populations without proprietary aid which would require a master list of players for a given game and all of their personal information (potential ethical issue).
It's a standing issue in the whole of online gaming studies. The double bias that comes from a self-selected sample and posting location is in fact a real thing--I won't deny that and I'll be the first one to admit it. It's a limitation and potential confound--this's a matter of fact but one that's hard to get around.
When you're speaking to multiple groups that reach hundreds of millions of players you have to pick and choose your battles. The best thing you can do is to sample from many different avenues: subreddits, forums, social media platforms, groups, word of mouth, collegiate systems, etc. That is the unfortunate nature of the beast.
Some of the only researchers that've had proprietary access to companies populations have been individuals like Scott Caplan (EQ) and Dimitri Williams (League of Legends and some Chinese MMO). Not an easy thing to do and something that runs risk with IRB approval because of potential conflicts of interest and ethical bylines concerning obtaining sensitive personal data.
Sampling like this typically results in several thousand participants across dozens of games. There is a degree of generalizability to the results but it does call into question proper external validity of findings. Sample homogeneity will almost always be a limitation until we find a means by which to promote better heterogeneity. A self-selected sample and statistical invalidity are not mutually exclusive things. Though one does have to be careful in the manner by which they speak about results.
Ultimately these studies are more like hourglasses. You start general, assess and understand groups, isolate populations to one of interest, then investigate those individuals more intimately and expand understand of those population segments. The tail-end of things is rooted in more traditional methods and avoids some of these problems while promoting others regarding sample size.
Now, speaking of sample size. Because the sample is so large the fact that the sample is self-selected and not based in probability sampling becomes a little less of an issue. If it was a self-selected floating around the smallest N required for statistical significance (~100) then it'd be a glaring issue.
The thing about these games is that their populations are relatively potpourri. If you spread your sample around enough even with self-selection you will wind up with individuals from all walks of life and of vastly different backgrounds. In my undergrad work I had ages 18-67. You wind up with a relatively vast representation of the population. Gender usually winds up split at approximately 80-20 for MMORPG crowds (waiting to see the MOBA end, seems to be vastly more male).
We'll never be able to hit on everyone in the population, it's simple far too widespread and diverse for that. So, as I mentioned earlier we've got to pick our battles, embrace what drawbacks come from the methods, and make the most out of what we find (with the limiting caveats regarding sampling methodologies in mind).
Also, math can be hard. However statistic programs do the bulk of the leg work for us (fortunately, right?). Though even that's not necessarily an easy thing to do. If you're curious about the analysis I do typically run, I tend to do a lot of structural equational modeling (full, nested, and pairwise path modeling) and multi-step hierarchal multiple regressions (I think my highest was something like 21 steps when I was in undergrad(?)) in my studies given the nature of what I tend to explore. It's not the run of the mill simple bivariates, t-test, (M)anova, regression, or cross tab analysis but it's what works.
Nothing (and nobody) is perfect, just doing the best I can with what I've got.
Hopefully this helps.
I look forward to hearing back from you and dialoguing further.
Best, Justin