Andrew Gelman is a seemingly tireless crusader against the sloppy use of p-values. Today he posted a very short (4 page) new article that explains some of the problems with p-values, and gives some quick examples of when they fall apart vs. when they merely do no harm. I recommend reading the whole thing, especially if you’ve recently been exposed to the standard two semester sequence of statistics in Sociology or econometrics. If you’re totally unfamiliar with Bayesian analysis, some of the terms will be a bit confusing, but it’s a good opportunity to search around a bit and get a feel for the language of Bayesianism. A couple gems:

The casual view of the P value as posterior probability of the truth of the null hypothesis is false and not even close to valid under any reasonable model, yet this misunderstanding persists even in high-stakes settings (as discussed, for example, by Greenland in 2011). The formal view of the P value as a probability conditional on the null is mathematically correct but typically irrelevant to research goals (hence, the popularity of alternative—if wrong—interpretations).

This passage, from the opening, names both the most common but wrong interpretation and identifies one source of that wrongness: what p-values actually mean is not very interesting, and so we’d much rather they mean what they don’t.

One big practical problem with P values is that they cannot easily be compared. … Consider a simple example of two independent experiments with estimates (standard error) of 25 (10) and 10 (10). The first experiment is highly statistically significant (two and a half standard errors away from zero, corresponding to a normal-theory P value of about 0.01) while the second is not significant at all. Most disturbingly here, the difference is 15 (14), which is not close to significant. The naive (and common) approach of summarizing an experiment by a P value and then contrasting results based on significance levels, fails here, in implicitly giving the imprimatur of statistical significance on a comparison that could easily be explained by chance alone.

Gelman has written about this example many times before under the heading “The difference between significant and not significant is not significant.” This is the quickest explanation I’ve seen.

Right now, I’m reading Mirowski’s (2011) Science-Mart: Privatizing American Science. The book is provocative, as we expect from Mirowski, and much less technical and more accessible than some of his earlier works. The basic argument is that neoliberalism ruined American science, and the American university (among other things). Neoliberals, with Hayek foremost among them, argue that The Market is the only really good source of information:

The Market is an artifact, but it is an ideal processor of information. Every successful economy is a knowledge economy. It knows more than any individual, and therefore it cannot be surpassed as a mechanism of coordination. (Mirowski 2011: 29)

Because The Market is, for neoliberals, the ultimate information processor, it makes sense that the production of information should also be thrown into the market. That is, scientific research, R&D, and all that should be privatized.

So far so good, I suppose. But along with his criticisms of neoliberalism, Mirowski simultaneously targets many of the most prominent scholars in science studies of the past 30 years (Latour, Shapin, Porter, etc.). Mirowski argues that science studies has been complicit in the neoliberal project, or at least that most* science studies research comes to neoliberal conclusions. For example, Mirwoski quotes and critiques a review by Ted Porter of John Carson’s book on the history of intelligence:

“We also are chipping away at the public image, indeed the self-conception of ‘intellectuals’ and scientists, who typically have argued in universals. These individuals speak and write of what should be valid for everyone, of what is true or just or good. Intellectual history and history of science aspire to undermine these universals by recovering their specificity and locality.” (Porter 2009, 642)

This could just have easily been written by Hayek himself; all one needs to add is an explicit appeal to the market test to supersede the discredited universals. (Mirowski 2011: 319)

Mirowski is not exactly arguing that science studies scholars hate science, but he is arguing that they are attempting to dethrone it in a way similar to Hayek by equating it to marketing, and accusing scientists of self-interested behavior. It’s worth noting, of course, that Porter does not add an “explicit appeal to the market test to supersede the discredit universals.” Mirowski is right to note that science studies scholars often take a critical stance towards scientific overreach – the famed criticism of scientists’ tendency to “speak from nowhere”, as if outside society (cf. Haraway 1991). But the point is not to undermine science the way that Mirowski claims neoliberalism wants to undermine science (and replace it with the very profitable manufacture of ignorance). The point, or at least one point, of science studies scholarship is to resituate science in its various localities precisely to make it stronger, at least stronger against certain types of criticism.**

One of the common moves in science studies*** is to show how science is produced by flesh-and-blood(-and-machine) people, who are potentially fallible, racist, sexist, self-interested, and more generally shaped by the prevailing institutions and ideas of their times (just as they are in turn shaping the ideas and institutions yet to come). For me, this emphasis on the how and the who of science, the reinserting of the active tense into scientific discourse, comes from a belief that science is most susceptible to critique when it is least understood.

An example might serve better than a generic explanation. Consider the “climate gate” email scandal. As a brief refresher, the Climatic Research Unit at East Anglia University had its email hacked, and released. Some of the emails led to accusations of scientific malfeasance because they discussed the manipulation of certain graphics to emphasize the argument of a paper. This supposed malfeasance was rhetorically employed to criticize the current consensus on climate change as one big hoax or conspiracy. But digging into the emails, all we actually see is flesh-and-blood-and-spreadsheet researchers trying to make their graphs look prettier to emphasize their research findings. Here’s the summary from Wikipedia****:

Many commentators quoted one email referring to “Mike’s Nature trick” which Jones used in a 1999 graph for the World Meteorological Organization, to deal with the well-discussed tree ring divergence problem “to hide the decline” that a particular proxy showed for modern temperatures after 1950, when measured temperatures were rising. These two phrases from the emails were also taken out of context by climate change sceptics including US Senator Jim Inhofe and former Governor of Alaska Sarah Palin as though they referred to a decline in measured global temperatures, even though they were written when temperatures were at a record high.[32] John Tierney, writing in the New York Times in November 2009, said that the claims by sceptics of “hoax” or “fraud” were incorrect, but the graph on the cover of a report for policy makers and journalists did not show these non-experts where proxy measurements changed to measured temperatures.[33] The final analyses from various subsequent inquiries concluded that in this context ‘trick’ was normal scientific or mathematical jargon for a neat way of handling data, in this case a statistical method used to bring two or more different kinds of data sets together in a legitimate fashion.[34][35] The EPA notes that in fact, the evidence shows that the research community was fully aware of these issues and was not hiding or concealing them.[36]

Of course a research paper is a form of rhetoric, produced by particular speakers trying to convince others of their conclusions. What else could it possibly be? But the fact that there is rhetoric in science doesn’t mean it’s wrong, a lie, a conspiracy, or a hoax. It simply means that science is the product of actual people (and their tools) and progresses through the back-and-forth of a dialog, not the uniform march towards enlightenment. Readers of Paul Edwards’ fabulous book on the history of climate data and climate models, A Vast Machine, learn about the immense effort that goes into our historical and contemporary knowledge of the climate. They would learn that there is no single measurement of the climate, that we have to cobble together thousands and thousands of diverse sources, from tree rings to ship’s logs to satellite imagery, to produce a uniform dataset. After reading A Vast Machine, the East Anglia emails seem commonplace: scientists working to make sense of a complicated piece of data insider a community of knowers. It’s only when we possess a tremendously lionized view of science-as-revealed-truth that we can have our faith shaken by seeing into how the sausage is made.

Similarly, returning to John Carson’s work on intelligence, good STS research should leave us skeptical, especially when the scientific tradition in question has such tight links to overt forms of social domination. In Carson’s case, the history of intelligence is inextricably bound to the history of racial hierarchies and more generally to the justification of inequality. The point here is not to say that IQs are “wrong”, but rather to inquire, what do they mean and what purposes have they served? What Carson debunks is not psychology, but rather America’s pretensions of meritocracy founded on a scientific form of discrimination. Understanding the positions of key figures in the history of intelligence testing – “recovering their specificity and locality” as Porter put it – helps us to understand how IQ ended up embroiled in this whole project, as well as the other purposes to which it has been usefully put.

In any case, I think Mirowski errs grievously when he collapses science studies’ rediscovery of science’s locality with Hayek’s worship of the market as the sole arbiter of truth. More generally, I think critics of science studies miss the mark when they imagine STS to be “anti-science.” At times, STS has indeed sounded like a Republican anti-science talking point (as Latour himself points out!). But that is not science studies’ project, and it never has been.

* Mirowski speaks favorably of, for example, Oreskes and Conway’s Merchants of Doubt, which makes somewhat similar arguments but with a specific focus on high-profile moments of the production of ignorance by anti-regulatory business interests (Tobacco Science, climate change skeptics, etc.).
** My thinking here is guided by Latour’s (2004) Why Has Critique Run Out of Steam?, where he asks and attempts to answer several related questions. Latour focuses on the similarities between STS arguments and the claims of conservatives in their war on science.
*** Note that I use science studies and STS interchangeably. STS stands for Science, Technology and Society, or Science and Technology Studies, and there have historically been tensions between the various terms one could apply to the field, but they are beyond the scope of this blog post and frankly I’ve never quite figured out what was at stake there anyway.
**** Another little gripe with Mirowski concerns his dislike of Wikipedia. What’s with that? I understand his skepticism towards the copyleft movement generally (which he argues is basically too little, a conclusion that Lawrence Lessig himself came to as well when he switched to working on political corruption and campaign finance as a root cause). But that’s no reason to hate Wikipedia, which is a pretty neat community, and produces incredibly useful (if imperfect) resources for free. All this could lead to another rant about how we need to teach our students how to use Wikipedia rather than simply enjoining them to never look at it (while simultaneously using it ourselves routinely).

Simplicity is often a virtue. We like to call this virtuous simplicity parsimony, and there are all sorts of reasons to think that parsimony is fundamental to defining useful knowledge. Borges’ “On Exactitude in Science” is a classic here, where Borges artfully shows in a single paragraph the worthlessness of a perfect map, since the map would be the size of the territory, and thus no aid in navigation. But what does simplicity look like in the sciences? Does it have just one form?

In today’s science studies workshop, we discussed Evelyn Fox Keller’s The Century of the Gene. The book traces the history of 20th century genetics, from the rediscovery of Mendel through the Human Genome Project, and shows how the concept of the gene was initially quite vague, but became much more focused following Watson and Crick’s publication of the structure of DNA, which seemed to have all the complex properties needed to be the material form of genes. Following advances in the 1970s-1990s, however, our notion of the gene has become fuzzy once again as we learned that not all DNA sequences code for proteins, and that there are hosts of mechanisms in place both inside DNA sequences and in the supporting infrastructure to regulate DNA activation, to detect and correct errors, and even to induce mutations. The book is short and totally free of (science studies) jargon, and seems to be aimed more at a general reader or perhaps undergraduate biology or history student than an STS crowd, but because of that it leaves a lot of material undertheorized (though simultaneously being a much easier quick afternoon read).

To simplify an already simplified story, one of the tensions in the book is between two notions of what a gene is: a kind of stuff (e.g. DNA sequences), a set of functions (the unit of transmission of heredity, a piece of the code for a complete organism). Fox Keller traces the history of these competing notions of genes back to just before the term itself was coined, in the early 20th century, and then follows them through to the end of the century. The expanded understanding of genetics produced by the late 20th century developments (debates over gene activation and junk DNA, etc.) burst open the exact linkage of 1 gene –> 1 protein, the tight connection between structure and function.

All of this is a fascinating refresher and update for someone whose last biology class was a decade ago in high school, around when this book was published. But I think the book also makes a number of points, implicitly, that are of relevance to STS scholars in other areas. One concerns the rise and fall in specificity of a term: “gene” was a useful term in the early 20th century because of its openness (genes are how organisms do heredity, whatever that ends up being), and was useful again in the mid-20th century for its concreteness (DNA = genes), but eventually must be reopened or abandoned to make more progress.

Another potentially useful generalization from the history of 20th century genetics concerns different understandings of a simple or parsimonious explanation. The debate referred to above between “structural” or material understandings of what genes are and functions understandings maps onto two different understandings of a simple explanation or theory, what I’ll call “ontological simplicity” and “mechanistic simplicity”. You can think of these as being tightly related to issues of “representation” vs. “intervention” in an Ian Hacking sense. An ontologically simple theory posits some basic blocks whose interactions explain the phenomenon of interest. So, atomic theory here would be the obvious parallel: all of chemistry and biology can be reduced to the interaction of a few basic atomic building blocks. Similar, one version of genetics is that organisms (phenotypes) can be explained in terms of the interactions of basic building blocks, genes, that have a common structure, DNA.

On the other hand, a mechanistically simple theory is one that focuses on the least intervention required to produce a given change, given a host of other (unspecified) factors that are held constant. So, mechanistically simple theories tell us that if you swap this bit of DNA with this other bit, you produce a fly with too many eyes, or a human being with Tay-Sachs. If you swap these other bits of DNA, or mess with the cell environment in another way, something entirely different happens. These theories (findings, phenomena) allow for a very complex world filled with junk DNA and gene-environment interactions, but one whose complexity can be bracketed for certain purposes. As Fox Keller describes, the success of the cloning of Dolly the sheep had more to do with scientists figuring out little tricks to get a mammalian ovum to behave rather than some deep underlying theory of why mammal and lizard ova behaved differently such that the latter were relatively easy to clone. The process worked, and worked repeatable, and thus produced a new phenomenon, a successful intervention.

These two notions of simplicity don’t necessarily work at cross purposes, and may indeed work best in iteration – come up with a new, parsimonious ontology, then tinker with it to produce new phenomenon in ways that bust up the ontology, repeat if possible. But the two are distinct, and it’s worth thinking about the ways that debates over parsimony in other fields (say economics) may map onto the same or similar categories and thus produce tensions when results are less clear. Parsimony is a virtue, but not always a simple one.

It’s Fall, and for academics, Fall is grant and fellowship season. In Sociology, the NSF is one of the biggest funders, especially the prestigious pre-dissertation Graduate Research Fellowship, and the equally prestigious Dissertation Improvement Grant. The NSF is a sometimes frustrating experience, especially for qualitative and historical researchers who feel like their “style” is disfavored by the structure of the proposals.* In particular, the NSF has a requirement that researchers address “falsifiability” – in other words, how might you be wrong? For qualitative and historical researchers, this formulation can be a bit alienating – if you are doing a project with a significant inductive component, it’s not easy to think in terms of how the data might disprove your theory.

I propose a slight reframing of these discussions for “non-canonical” researchers.** Latour famously defined reality as “that which resists.” Or, more specifically, “that which cannot be changed at will.” In some sense, the NSF is (reasonably) defining “science” as investigation which has the potential to be wrong, which offers the possibility that reality will resist. To do science is to be open to unexpected resistances.*** Reframed this way, instead of asking, “how is this falsifiable?” we can ask, “where might reality intrude unexpectedly in your story? What are you doing to seek out possible resistance to your claims?”

This formulation abandons some of the implicit, and outdated, philosophy of science that encircle the term “falsifiability” while still maintaining a very reasonable standard that scientific work must have the potential of being resisted, and must be open to that resistance.

* For some interesting debates on the subject, see the NSF report on getting funded to do qualitative research, Howard Beckers’ excellent response “How to Find Out How to Do Qualitative Research”, both of which are discussed on Scatterplot here.
** A term I borrow from Luker’s excellent methods book, Salsa Dancing into the Social Sciences to describe (generally) qualitative and historical research that is more interpretive, meaning, context and history oriented.
*** A bit tangentially, but I wonder if this formulation might also help modern science studies scholars in their sometimes difficult quest to distinguish “tobacco science” from public health, or climate change skeptics from legitimate climatology (see, for example, Agnotology).