Following the blog rabbit hole today, I came across an interesting statistics and data analysis blog I hadn’t seen before: Simply Statistics. The blog authors are biostatisticians at Johns Hopkins, and at least one is creating a 9-month MOOC sequence on data analysis that looks quite interesting. So, far my favorite post (and the one that led me to the blog) is a counter-rant to all the recent p-value bashing (e.g. this Nature piece): On the scalability of statistical procedures: why the p-value bashers just don’t get it. The post’s argument boils down to something like, “P-values, there is no alternative!” But check out the full post for the interesting defense of the oft-maligned and even more oft-misinterpreted mainstay of conventional quantitative research.
Apart from that post, I also enjoyed a link to a recent working paper, which is what I wanted to highlight here. Even though the blog authors defend p-valus as a simple way of controlling researcher degrees of freedom, they also seem to be part of a growing group of statisticians interested in finding ways of correcting for the “statistical significance filter“, as Andrew Gelman puts it. The method presented in “P-Curve Fixes Publication Bias: Obtaining Unbiased Effect Size Estimates from Published Studies Alone” seems quite intuitive. Basically, the authors show how to simulate a p-curve (distribution of p-values) that best matches the observed p-values in a collection of studies, given the assumption that only significant results are published (but not perfectly accounting for other forms of p-hacking, discussed in the paper). Although the paper is short, it presents payoffs for analysis of two vexing problems, including the relationship between unemployment and the minimum wage. Here’s the example reproduced in full:
Our first example involves the well-known economics prediction that increases in minimum wage raise unemployment. In a meta-analysis of the empirical evidence, Card and Krueger (1995) noted that effect size estimates are smaller in studies with larger samples and comment that “the studies in the literature have been affected by specification-searching and publication biases, induced by editors’ and authors’ tendencies to look for negative and statistically significant estimates of the employment effect of the minimum wage […] researchers may have to temper the inferences they draw […]” (p.242).
From Figure 1 in their article (Card & Krueger, 1995) we obtained the t-statistic and degrees of freedom from the fifteen studies they reviewed. As we show in our Figure 4, averaging the reported effect size estimates one obtains a notable effect size, but correcting for selective reporting via p-curve brings it to zero. This does not mean increases in minimum wage would never increase unemployment, it does mean that the evidence Card and Kruger collected suggesting it had done so in the past, can be fully accounted by selective reporting. P-curve provides a quantitative calibration to Card and Krueger’s qualitative concerns. The at the time controversial claim that the existing evidence pointed to an effect size smaller than believed was not controversial enough; the evidence actually pointed to a nonexisting effect.
So, Nelson et al. provide an intuitive way of formalizing Card & Krueger’s assertion that publication bias could account for some of the findings of a negative relationship between unemployment and minimum wage increases – and even further, that publication bias could actually reduce the best estimate of the effect to zero (which seems consistent with much, thought certainly not all, of the recent literature).
These methods seem really neat, but I’m not entirely sure what problems in sociology we could generalize them to. In the subfields I follow most closely, most research is either not quantitative, or is based on somewhat idiosyncratic data and hence it’s hard to imagine a bunch of studies with sufficiently comparable dependent variables and hypotheses from which one could draw a distribution. I’d bet demographers would have more luck. But in economic sociology, published replication seems sufficiently rare to prevent us from making much headway on the the issue of publication bias using quantitative techniques like this – which perhaps points to a very different set of problems.