John Ioannidis is an increasingly prominent epidemiologist known primarily for his debunking-style papers on the problems of health research. In his 2005 “Why Most Published Research Findings Are False”, Ioannidis argued that most statistical research relies on terrible practices that suggest that false positives likely exceed true positives. Ioannidis has applied this argument most thoroughly to genetic association studies, which search for a correlation between a large collection of potential genes and a given outcome.

Just recently, however, Ioannidis and co-author Doucouliagos have turned this same analytical apparatus on empirical economics research. Although the paper is short on detailed analysis, the overall take presented is pretty damning. Here’s the abstract, and the quote of the day:

The scientific credibility of economics is itself a scientific question that can be addressed with both theoretical speculations and empirical data. In this review, we examine the major parameters that are expected to affect the credibility of empirical economics: sample size, magnitude of pursued effects, number and pre-selection of tested relationships, flexibility and lack of standardization in designs, definitions, outcomes and analyses, financial and other interests and prejudices, and the multiplicity and fragmentation of efforts. We summarize and discuss the empirical evidence on the lack of a robust reproducibility culture in economics and business research, the prevalence of potential publication and other selective reporting biases, and other failures and biases in the market of scientific information. Overall, the credibility of the economics literature is likely to be modest or even low. [emphasis added]

Oh, snap. Their preferred solutions seem to be similar to the recommendations for other statistical sciences, including better meta-analysis, more replication, and so on. They are also enthusiastic about RCTs without really noting how RCTs are only appropriate for answering some questions within the bailiwick of economics, and are relatively limited in their usefulness for answering other, very important questions. Anyway, recommended if you’re into critiques of economics research and best research practices.

Scary thought: what would this paper look like if the target was not empirical economics but rather quantitative sociology?