Gelman’s Problems with P-Values

Andrew Gelman is a seemingly tireless crusader against the sloppy use of p-values. Today he posted a very short (4 page) new article that explains some of the problems with p-values, and gives some quick examples of when they fall apart vs. when they merely do no harm. I recommend reading the whole thing, especially if you’ve recently been exposed to the standard two semester sequence of statistics in Sociology or econometrics. If you’re totally unfamiliar with Bayesian analysis, some of the terms will be a bit confusing, but it’s a good opportunity to search around a bit and get a feel for the language of Bayesianism. A couple gems:

The casual view of the P value as posterior probability of the truth of the null hypothesis is false and not even close to valid under any reasonable model, yet this misunderstanding persists even in high-stakes settings (as discussed, for example, by Greenland in 2011). The formal view of the P value as a probability conditional on the null is mathematically correct but typically irrelevant to research goals (hence, the popularity of alternative—if wrong—interpretations).

This passage, from the opening, names both the most common but wrong interpretation and identifies one source of that wrongness: what p-values actually mean is not very interesting, and so we’d much rather they mean what they don’t.

One big practical problem with P values is that they cannot easily be compared. … Consider a simple example of two independent experiments with estimates (standard error) of 25 (10) and 10 (10). The first experiment is highly statistically significant (two and a half standard errors away from zero, corresponding to a normal-theory P value of about 0.01) while the second is not significant at all. Most disturbingly here, the difference is 15 (14), which is not close to significant. The naive (and common) approach of summarizing an experiment by a P value and then contrasting results based on significance levels, fails here, in implicitly giving the imprimatur of statistical significance on a comparison that could easily be explained by chance alone.

Gelman has written about this example many times before under the heading “The difference between significant and not significant is not significant.” This is the quickest explanation I’ve seen.

Advertisements

The Left’s (Non-) War on Science: Anti-Nuclear and Anti-Hydroelectric are not “Anti-Science”

This week, Scientific American has an interesting, but I think flawed, article about the anti-science attitudes on the left. More specifically, Michael Shermer argues that:

Whereas conservatives obsess over the purity and sanctity of sex, the left’s sacred values seem fixated on the environment, leading to an almost religious fervor over the purity and sanctity of air, water and especially food.

Shermer’s starting point is the admittedly frightening statistic that 41% of Democrats (along with 58% of Republicans) are young Earth Creationists, and 19% of Democrats (compared to 51% of Republicans) doubt that the Earth is warming. That last difference is quite huge, and has been much discussed in the sociological literature (e.g. McCright and Dunlap 2011). Similarly, general trust in science is now substantially higher among liberals than conservatives (Gauchat 2012). So, there may be substantial anti-science sentiment on left, but there is not nearly as much as there is on the right.

That said, Shermer’s more interesting claim is not that the left is as strongly anti-science as the right, but rather that it’s anti-science in different ways. And here, I think his examples are just off or misleading or involve some sort of redefinition of what it means to be “anti-science.” Here’s one claim from the article, citing a recent book:

There is more, and recent, antiscience fare from far-left progressives, documented in the 2012 book Science Left Behind (PublicAffairs) by science journalists Alex B. Berezow and Hank Campbell, who note that “if it is true that conservatives have declared a war on science, then progressives have declared Armageddon.” On energy issues, for example, the authors contend that progressive liberals tend to be antinuclear because of the waste-disposal problem, anti–fossil fuels because of global warming, antihydroelectric because dams disrupt river ecosystems, and anti–wind power because of avian fatalities. The underlying current is “everything natural is good” and “everything unnatural is bad.”

Although I agree that this characterization of the left’s attitude towards nature has something going for it (especially as contrasted with the right), it’s not really an attitude about science. It’s about the moral weight of different kinds of harms. Specifically, it’s not “anti-science” to oppose an increased use of nuclear energy because of concerns about storing nuclear waste, nor to oppose hydroelectric power because it disrupts ecosystems. At least, it’s not anti-science the same way that denying global warming is anti-science: nuclear waste actually is a significant problem, and dams do actually disrupt ecosystems. Whether or not these costs are worth the payoffs (i.e. reduced fossil fuel usage) is a very different question than whether or not these costs exist.

It makes sense to read questions like “How old is the Earth?” or “Has the average temperature of the Earth increased in the past century?” as connected to an underlying attitude about science because these are issues which scientists have made strong positive claims about: the Earth is about 4.5 billion years old, the Earth warmed about .6 degrees Celsius in the 20th century. To deny these claims is to deny science. But to claim that we undervalue the disruption of ecosystems by dams, for example, is not an “anti-science” claim.*

I’m not sure exactly what’s going on with Shermer’s article, except perhaps the contrary urge (we all know the Right is so much worse on science, but look, really the Left is just as bad!), but it seems to me like a classic example of false equivalence, powered by the conflation of skepticism towards technology on the grounds that it may have unforeseen consequences and the rejection of science. In sum, I remain a skeptic about the existence of the “Liberals’ War on Science.”

* Which is not to say that there aren’t tons of claims made by opponents of nuclear power or dams that run contrary to nuclear science, ecology, and so on. I’m sure there are. But that doesn’t amount to a campaign to delegitimize science as an institution: what we would properly call a “War on Science.” But maybe that’s the definitional issue that’s precisely troubling me, and that is entirely implicit in the article: what does it mean to be anti-science in the first place?

Climate Skeptics and the OJ Simpson Defense

Predictably, the NOAA announcement that 2012 was the hottest year on record in the US (a full degree higher than the previous record high average, and three degrees higher than the 20th century average) has brought out the climate skeptics in force. Fox News (surprise!) ran this story: Hottest year ever? Skeptics question revisions to climate data. The details are not especially important: the NOAA re-ran some of its models and updated its historical data series from version 2.0 to 2.5, this apparently caused some slight increase in measured warming in the late 20th century. These changes are pretty small in comparison to the massively hot 2012, however. The defenders of the NOAA – and, let’s say, people trying to keep humanity from frying itself to death – tried to make this argument but in so doing offer an interesting if pessimistic analogy:

Aaron Huertas, a spokesman for the Union of Concerned Scientists, argued that the debate over the adjustments misses the bigger picture.

“Since we broke the [temperature] record by a full degree Fahrenheit this year, the adjustments are relatively minor in comparison,”

“I think climate contrarians are doing what Johnny Cochran did for O.J. Simpson — finding anything to object to, even if it obscures the big picture. It’s like they keep finding new ways to say the ‘glove doesn’t fit’ while ignoring the DNA evidence.”

I think this analogy is pessimistic – if incredibly accurate – because Cochrane managed to convince the relevant decision-makers (the jury) that there was reasonable doubt about OJ’s guilt, even as people still routinely assume OJ was guilty (as Huerta implicitly does in the quote). Let’s hope the analogy fails as a prediction. So far though, it seems spot on.

Hyperscience QOTD: Shapin on Pseudoscience

Eminent historian of science Steven Shapin has a review essay in the London Review of Books about a new history of pseudoscience (specifically, the Velikovsky affair). The end of the essay is especially brilliant, as Shapin thinks through what we can learn from the affair about the more general problem of demarcating pseudoscience from the real stuff (whatever that might be):

Whenever the accusation of pseudoscience is made, or wherever it is anticipated, its targets commonly respond by making elaborate displays of how scientific they really are. Pushing the weird and the implausible, they bang on about scientific method, about intellectual openness and egalitarianism, about the vital importance of seriously inspecting all counter-instances and anomalies, about the value of continual scepticism, about the necessity of replicating absolutely every claim, about the lurking subjectivity of everybody else. Call this hyperscience, a claim to scientific status that conflates the PR of science with its rather more messy, complicated and less than ideal everyday realities and that takes the PR far more seriously than do its stuck-in-the-mud orthodox opponents. Beware of hyperscience. It can be a sign that something isn’t kosher. A rule of thumb for sound inference has always been that if it looks like a duck, swims like a duck and quacks like a duck, then it probably is a duck. But there’s a corollary: if it struts around the barnyard loudly protesting that it’s a duck, that it possesses the very essence of duckness, that it’s more authentically a duck than all those other orange-billed, web-footed, swimming fowl, then you’ve got a right to be suspicious: this duck may be a quack.

I wonder if economics, and perhaps social science more generally, suffer from a kind of generalized case of hyperscience. Think, for example, of economists’ much more explicit invocation of philosophy of science (well, a particular version of it) to justify their practices. It reminds me a lot of Shapin’s definition of hyperscience “a claim to scientific status that conflates the PR of science with its rather more messy, complicated and less than ideal everyday realities and that takes the PR far more seriously than do its stuck-in-the-mud orthodox opponents.” But think also of the constant refrains of testing hypotheses in major sociology journals, and all the hemming and hawing about methods. Are we protesting too much? Or are things just different in the social sciences, trapped as we are in the nether regions between objectivity and advocacy, causal inference and normative theory, etc.? What would our criteria be for identifying social pseudoscience or hyperscience?

“Bad Pharma” QotD

Ben Goldacre writes a very cool, if frequently depressing, blog Bad Science. He’s also written a new book about Bad Science in Big Pharma, appropriately titled Bad Pharma. He posted the foreword on his blog today, including the following paragraph which summarizes the entire argument of the book:

Drugs are tested by the people who manufacture them, in poorly designed trials, on hopelessly small numbers of weird, unrepresentative patients, and analysed using techniques which are flawed by design, in such a way that they exaggerate the benefits of treatments. Unsurprisingly, these trials tend to produce results that favour the manufacturer. When trials throw up results that companies don’t like, they are perfectly entitled to hide them from doctors and patients, so we only ever see a distorted picture of any drug’s true effects. Regulators see most of the trial data, but only from early on in its life, and even then they don’t give this data to doctors or patients, or even to other parts of government. This distorted evidence is then communicated and applied in a distorted fashion. In their forty years of practice after leaving medical school, doctors hear about what works through ad hoc oral traditions, from sales reps, colleagues or journals. But those colleagues can be in the pay of drug companies – often undisclosed – and the journals are too. And so are the patient groups. And finally, academic papers, which everyone thinks of as objective, are often covertly planned and written by people who work directly for the companies, without disclosure. Sometimes whole academic journals are even owned outright by one drug company. Aside from all this, for several of the most important and enduring problems in medicine, we have no idea what the best treatment is, because it’s not in anyone’s financial interest to conduct any trials at all. These are ongoing problems, and although people have claimed to fix many of them, for the most part, they have failed; so all these problems persist, but worse than ever, because now people can pretend that everything is fine after all.

I haven’t read the book yet, but if it’s anything like the blog, it will be compelling, well-researched, and incredibly infuriating.

Does Science Studies Hate Science? No.

Right now, I’m reading Mirowski’s (2011) Science-Mart: Privatizing American Science. The book is provocative, as we expect from Mirowski, and much less technical and more accessible than some of his earlier works. The basic argument is that neoliberalism ruined American science, and the American university (among other things). Neoliberals, with Hayek foremost among them, argue that The Market is the only really good source of information:

The Market is an artifact, but it is an ideal processor of information. Every successful economy is a knowledge economy. It knows more than any individual, and therefore it cannot be surpassed as a mechanism of coordination. (Mirowski 2011: 29)

Because The Market is, for neoliberals, the ultimate information processor, it makes sense that the production of information should also be thrown into the market. That is, scientific research, R&D, and all that should be privatized.

So far so good, I suppose. But along with his criticisms of neoliberalism, Mirowski simultaneously targets many of the most prominent scholars in science studies of the past 30 years (Latour, Shapin, Porter, etc.). Mirowski argues that science studies has been complicit in the neoliberal project, or at least that most* science studies research comes to neoliberal conclusions. For example, Mirwoski quotes and critiques a review by Ted Porter of John Carson’s book on the history of intelligence:

“We also are chipping away at the public image, indeed the self-conception of ‘intellectuals’ and scientists, who typically have argued in universals. These individuals speak and write of what should be valid for everyone, of what is true or just or good. Intellectual history and history of science aspire to undermine these universals by recovering their specificity and locality.” (Porter 2009, 642)

This could just have easily been written by Hayek himself; all one needs to add is an explicit appeal to the market test to supersede the discredited universals. (Mirowski 2011: 319)

Mirowski is not exactly arguing that science studies scholars hate science, but he is arguing that they are attempting to dethrone it in a way similar to Hayek by equating it to marketing, and accusing scientists of self-interested behavior. It’s worth noting, of course, that Porter does not add an “explicit appeal to the market test to supersede the discredit universals.” Mirowski is right to note that science studies scholars often take a critical stance towards scientific overreach – the famed criticism of scientists’ tendency to “speak from nowhere”, as if outside society (cf. Haraway 1991). But the point is not to undermine science the way that Mirowski claims neoliberalism wants to undermine science (and replace it with the very profitable manufacture of ignorance). The point, or at least one point, of science studies scholarship is to resituate science in its various localities precisely to make it stronger, at least stronger against certain types of criticism.**

One of the common moves in science studies*** is to show how science is produced by flesh-and-blood(-and-machine) people, who are potentially fallible, racist, sexist, self-interested, and more generally shaped by the prevailing institutions and ideas of their times (just as they are in turn shaping the ideas and institutions yet to come). For me, this emphasis on the how and the who of science, the reinserting of the active tense into scientific discourse, comes from a belief that science is most susceptible to critique when it is least understood.

An example might serve better than a generic explanation. Consider the “climate gate” email scandal. As a brief refresher, the Climatic Research Unit at East Anglia University had its email hacked, and released. Some of the emails led to accusations of scientific malfeasance because they discussed the manipulation of certain graphics to emphasize the argument of a paper. This supposed malfeasance was rhetorically employed to criticize the current consensus on climate change as one big hoax or conspiracy. But digging into the emails, all we actually see is flesh-and-blood-and-spreadsheet researchers trying to make their graphs look prettier to emphasize their research findings. Here’s the summary from Wikipedia****:

Many commentators quoted one email referring to “Mike’s Nature trick” which Jones used in a 1999 graph for the World Meteorological Organization, to deal with the well-discussed tree ring divergence problem “to hide the decline” that a particular proxy showed for modern temperatures after 1950, when measured temperatures were rising. These two phrases from the emails were also taken out of context by climate change sceptics including US Senator Jim Inhofe and former Governor of Alaska Sarah Palin as though they referred to a decline in measured global temperatures, even though they were written when temperatures were at a record high.[32] John Tierney, writing in the New York Times in November 2009, said that the claims by sceptics of “hoax” or “fraud” were incorrect, but the graph on the cover of a report for policy makers and journalists did not show these non-experts where proxy measurements changed to measured temperatures.[33] The final analyses from various subsequent inquiries concluded that in this context ‘trick’ was normal scientific or mathematical jargon for a neat way of handling data, in this case a statistical method used to bring two or more different kinds of data sets together in a legitimate fashion.[34][35] The EPA notes that in fact, the evidence shows that the research community was fully aware of these issues and was not hiding or concealing them.[36]

Of course a research paper is a form of rhetoric, produced by particular speakers trying to convince others of their conclusions. What else could it possibly be? But the fact that there is rhetoric in science doesn’t mean it’s wrong, a lie, a conspiracy, or a hoax. It simply means that science is the product of actual people (and their tools) and progresses through the back-and-forth of a dialog, not the uniform march towards enlightenment. Readers of Paul Edwards’ fabulous book on the history of climate data and climate models, A Vast Machine, learn about the immense effort that goes into our historical and contemporary knowledge of the climate. They would learn that there is no single measurement of the climate, that we have to cobble together thousands and thousands of diverse sources, from tree rings to ship’s logs to satellite imagery, to produce a uniform dataset. After reading A Vast Machine, the East Anglia emails seem commonplace: scientists working to make sense of a complicated piece of data insider a community of knowers. It’s only when we possess a tremendously lionized view of science-as-revealed-truth that we can have our faith shaken by seeing into how the sausage is made.

Similarly, returning to John Carson’s work on intelligence, good STS research should leave us skeptical, especially when the scientific tradition in question has such tight links to overt forms of social domination. In Carson’s case, the history of intelligence is inextricably bound to the history of racial hierarchies and more generally to the justification of inequality. The point here is not to say that IQs are “wrong”, but rather to inquire, what do they mean and what purposes have they served? What Carson debunks is not psychology, but rather America’s pretensions of meritocracy founded on a scientific form of discrimination. Understanding the positions of key figures in the history of intelligence testing – “recovering their specificity and locality” as Porter put it – helps us to understand how IQ ended up embroiled in this whole project, as well as the other purposes to which it has been usefully put.

In any case, I think Mirowski errs grievously when he collapses science studies’ rediscovery of science’s locality with Hayek’s worship of the market as the sole arbiter of truth. More generally, I think critics of science studies miss the mark when they imagine STS to be “anti-science.” At times, STS has indeed sounded like a Republican anti-science talking point (as Latour himself points out!). But that is not science studies’ project, and it never has been.

* Mirowski speaks favorably of, for example, Oreskes and Conway’s Merchants of Doubt, which makes somewhat similar arguments but with a specific focus on high-profile moments of the production of ignorance by anti-regulatory business interests (Tobacco Science, climate change skeptics, etc.).
** My thinking here is guided by Latour’s (2004) Why Has Critique Run Out of Steam?, where he asks and attempts to answer several related questions. Latour focuses on the similarities between STS arguments and the claims of conservatives in their war on science.
*** Note that I use science studies and STS interchangeably. STS stands for Science, Technology and Society, or Science and Technology Studies, and there have historically been tensions between the various terms one could apply to the field, but they are beyond the scope of this blog post and frankly I’ve never quite figured out what was at stake there anyway.
**** Another little gripe with Mirowski concerns his dislike of Wikipedia. What’s with that? I understand his skepticism towards the copyleft movement generally (which he argues is basically too little, a conclusion that Lawrence Lessig himself came to as well when he switched to working on political corruption and campaign finance as a root cause). But that’s no reason to hate Wikipedia, which is a pretty neat community, and produces incredibly useful (if imperfect) resources for free. All this could lead to another rant about how we need to teach our students how to use Wikipedia rather than simply enjoining them to never look at it (while simultaneously using it ourselves routinely).

The Climate Change Dystopia and National Accounts

I joke a lot about the robot apocalypse. I follow the news on the latest things we’ve taught robots and AI to do (e.g. consume organic matter, lie, shoot guns, recognize cats, etc.) and laugh at how when you add their capabilities together, you get the robot apocalypse that fiction (and Charli Carpenter) have warned us about for years. I like to think about the robot apocalypse because, while plausible in a science fiction way, it doesn’t seem imminent. Unlike the Climate Change Dystopia (CCD, let’s call it).*

This morning I read a piece from the next issue of Rolling Stone, Global Warming’s Terrifying New Math. There’s not that much that’s truly new about the piece, but it does an excellent job of summarizing how bad things have gotten and how much worse they’re going to get absent immediate, radical changes. The piece is framed around three numbers: 2 degrees Celsius (the amount of warming scientists used to think would be allowable without triggering CCD, though new estimates are a bit lower), 565 gigatons of carbon (how much more we can dump into the atmosphere and keep global warming to 2 degrees Celsisus), and 2,795 gigatons of carbon – the amount of fossil fuels that large companies already have in their reserves. That’s perhaps the most novel contribution of the article: to think through the political and economic consequences of the fact that fossil fuel producers have already discovered five times more fossil fuels than we can safely burn without triggering a catastrophe. These reserves are built into the value of large oil and gas companies around the world. These companies will fight tooth and nail for the right to burn all of what they’ve found, and even to search for more. In fact, they are already fighting hard, and have successfully produced doubt in the public about the extent and severity of the problem, as well as the scientific consensus (see Oreskes and Conway for details). And that’s going to destroy the world as we know it.

Here’s just one choice quote from the article on the absurd politics of energy and climate change:

Sometimes the irony is almost Borat-scale obvious: In early June, Secretary of State Hillary Clinton traveled on a Norwegian research trawler to see firsthand the growing damage from climate change. “Many of the predictions about warming in the Arctic are being surpassed by the actual data,” she said, describing the sight as “sobering.” But the discussions she traveled to Scandinavia to have with other foreign ministers were mostly about how to make sure Western nations get their share of the estimated $9 trillion in oil (that’s more than 90 billion barrels, or 37 gigatons of carbon) that will become accessible as the Arctic ice melts. Last month, the Obama administration indicated that it would give Shell permission to start drilling in sections of the Arctic.

For one clear demonstration of how things are going to get worse: this year’s record heats have made it harder to grow corn. When it’s too hot for too long, corn kernels don’t develop right, and they won’t produce corn. The US exports a tremendous amount of corn, and that keeps global food prices low (lower than they would be). This year, food prices are going to go up. Here’s a graph and caption from Paul Krugman:

I’ve been searching for something useful to say about the epic heat wave and drought afflicting U.S. agriculture, other than that this is the shape of things to come. Of course it’s about climate change: a rising number of temperature records is exactly what you’d expect given an underlying upward trend in global temperatures. And the economic consequences will be large: maybe 1 percent on U.S. consumer prices, but suffering and food riots in poorer nations that spend more of their income on food.

I don’t have too much insight into climate change beyond the apocalyptic things I read from the actual experts. The one point that arises from my research is that if we took climate change – or really, the environment at all – into account in our estimates of economic growth, the picture we have of “the economy” would look dramatically different. For example, Muller, Mendelsohn and Nordhaus (2011) produce a modified set of national accounts that take into account just one kind of environmental damage: air pollution (including estimates of the cost of CO2 emissions, where possible). Muller et al find that including this one form of pollution radically alters our assessment of how valuable certain industries are to the overall economy (understood here as “what GDP measures”).

For some industries (sewage treatment plants, solid waste combustion, stone quarrying, marinas, and petroleum fired and coal-fired power generation), [Gross Environmental Damage, GED] actually exceeds conventionally measured [Value Added, VA]. Crop and livestock production also have high GED/VA ratios, which is surprising given that these activities generally occur in rural (low marginal damage) areas. Other industries with high GED/VA ratios include water transportation, carbon black manufacturing, steam heat and air conditioning supply, and sugarcane mills. It is likely that many of these sources are underregulated.

Muller et al are cautious because of the many kinds of uncertainties in their estimates, and because GDP is already a wacky enough measure to begin with that it’s hard to take too seriously as a measure of economic welfare (despite the fact that we do just that all the time), but the point is clear: we are much less well off than we think if we think hard about pollution, and some industries may be downright destructive (at least at the margin), with their marginal product being less valuable than their marginal environmental damage.

Muller et al end with a call for the production official national accounts that take into account environmental damage. While they don’t make this link, I would argue that such accounts would be a helpful tool for climate change debates: every time someone asks, “But what will this do to the economy?” we would be able to provide a very different answer, one that recognized the previously undercounted costs (including increasing global warming). But, like most of the incremental political tools mentioned in the Rolling Stone article, these sorts of changes would likely take years, and only have small effects on our national debates. And we simply may not have the time for incremental measures anymore.

* For an excellent fictional account of the CCD, see Paolo Bacigalupi’s work, especially The Windup Girl.

American Science on Feathered Dinosaurs

American Science is a group blog written by a historians of science that I just came across. It looks excellent! A recent post discusses the controversy over feathered dinosaurs and whether or not Ian Hacking’s ideas about dynamic nominalism apply to fossils:

You might ask yourself (as I often do): why were feathered dinosaurs (of the non-avian variety) not discovered until so recently? Prior to the discovery announced in Nature today, you might have said: perhaps because feathered dinosaurs are relatively modest in terms of their size and appearance. (Modest until you have the tools to reconstruct their plumage pattern, that is!) But as I’ve already noted, part of what makes this discovery significant is that Y. huali is a close relative of T. rex. The next obvious question, to my mind, is this: if big, impressive, therapod dinosaurs like Y. huali had feathers, why didn’t anyone notice until now? Is it because paleontologists have only begin to research the evolution of dinosaurs in China, which is where feathered dinosaurs tend to be found, relatively recently? Or is it because paleontologists simply weren’t looking for feathered dinosaurs during the early 20th century, when the Western United States was understood to harbor the world’s richest dinosaur quarries?

Highly recommended, and I look forward to glancing through their older posts as well.

Two Versions of Simplicity in Science: Structural vs. Functional Genetics

Simplicity is often a virtue. We like to call this virtuous simplicity parsimony, and there are all sorts of reasons to think that parsimony is fundamental to defining useful knowledge. Borges’ “On Exactitude in Science” is a classic here, where Borges artfully shows in a single paragraph the worthlessness of a perfect map, since the map would be the size of the territory, and thus no aid in navigation. But what does simplicity look like in the sciences? Does it have just one form?

In today’s science studies workshop, we discussed Evelyn Fox Keller’s The Century of the Gene. The book traces the history of 20th century genetics, from the rediscovery of Mendel through the Human Genome Project, and shows how the concept of the gene was initially quite vague, but became much more focused following Watson and Crick’s publication of the structure of DNA, which seemed to have all the complex properties needed to be the material form of genes. Following advances in the 1970s-1990s, however, our notion of the gene has become fuzzy once again as we learned that not all DNA sequences code for proteins, and that there are hosts of mechanisms in place both inside DNA sequences and in the supporting infrastructure to regulate DNA activation, to detect and correct errors, and even to induce mutations. The book is short and totally free of (science studies) jargon, and seems to be aimed more at a general reader or perhaps undergraduate biology or history student than an STS crowd, but because of that it leaves a lot of material undertheorized (though simultaneously being a much easier quick afternoon read).

To simplify an already simplified story, one of the tensions in the book is between two notions of what a gene is: a kind of stuff (e.g. DNA sequences), a set of functions (the unit of transmission of heredity, a piece of the code for a complete organism). Fox Keller traces the history of these competing notions of genes back to just before the term itself was coined, in the early 20th century, and then follows them through to the end of the century. The expanded understanding of genetics produced by the late 20th century developments (debates over gene activation and junk DNA, etc.) burst open the exact linkage of 1 gene –> 1 protein, the tight connection between structure and function.

All of this is a fascinating refresher and update for someone whose last biology class was a decade ago in high school, around when this book was published. But I think the book also makes a number of points, implicitly, that are of relevance to STS scholars in other areas. One concerns the rise and fall in specificity of a term: “gene” was a useful term in the early 20th century because of its openness (genes are how organisms do heredity, whatever that ends up being), and was useful again in the mid-20th century for its concreteness (DNA = genes), but eventually must be reopened or abandoned to make more progress.

Another potentially useful generalization from the history of 20th century genetics concerns different understandings of a simple or parsimonious explanation. The debate referred to above between “structural” or material understandings of what genes are and functions understandings maps onto two different understandings of a simple explanation or theory, what I’ll call “ontological simplicity” and “mechanistic simplicity”. You can think of these as being tightly related to issues of “representation” vs. “intervention” in an Ian Hacking sense. An ontologically simple theory posits some basic blocks whose interactions explain the phenomenon of interest. So, atomic theory here would be the obvious parallel: all of chemistry and biology can be reduced to the interaction of a few basic atomic building blocks. Similar, one version of genetics is that organisms (phenotypes) can be explained in terms of the interactions of basic building blocks, genes, that have a common structure, DNA.

On the other hand, a mechanistically simple theory is one that focuses on the least intervention required to produce a given change, given a host of other (unspecified) factors that are held constant. So, mechanistically simple theories tell us that if you swap this bit of DNA with this other bit, you produce a fly with too many eyes, or a human being with Tay-Sachs. If you swap these other bits of DNA, or mess with the cell environment in another way, something entirely different happens. These theories (findings, phenomena) allow for a very complex world filled with junk DNA and gene-environment interactions, but one whose complexity can be bracketed for certain purposes. As Fox Keller describes, the success of the cloning of Dolly the sheep had more to do with scientists figuring out little tricks to get a mammalian ovum to behave rather than some deep underlying theory of why mammal and lizard ova behaved differently such that the latter were relatively easy to clone. The process worked, and worked repeatable, and thus produced a new phenomenon, a successful intervention.

These two notions of simplicity don’t necessarily work at cross purposes, and may indeed work best in iteration – come up with a new, parsimonious ontology, then tinker with it to produce new phenomenon in ways that bust up the ontology, repeat if possible. But the two are distinct, and it’s worth thinking about the ways that debates over parsimony in other fields (say economics) may map onto the same or similar categories and thus produce tensions when results are less clear. Parsimony is a virtue, but not always a simple one.

Falsifiability, Latour-style

It’s Fall, and for academics, Fall is grant and fellowship season. In Sociology, the NSF is one of the biggest funders, especially the prestigious pre-dissertation Graduate Research Fellowship, and the equally prestigious Dissertation Improvement Grant. The NSF is a sometimes frustrating experience, especially for qualitative and historical researchers who feel like their “style” is disfavored by the structure of the proposals.* In particular, the NSF has a requirement that researchers address “falsifiability” – in other words, how might you be wrong? For qualitative and historical researchers, this formulation can be a bit alienating – if you are doing a project with a significant inductive component, it’s not easy to think in terms of how the data might disprove your theory.

I propose a slight reframing of these discussions for “non-canonical” researchers.** Latour famously defined reality as “that which resists.” Or, more specifically, “that which cannot be changed at will.” In some sense, the NSF is (reasonably) defining “science” as investigation which has the potential to be wrong, which offers the possibility that reality will resist. To do science is to be open to unexpected resistances.*** Reframed this way, instead of asking, “how is this falsifiable?” we can ask, “where might reality intrude unexpectedly in your story? What are you doing to seek out possible resistance to your claims?”

This formulation abandons some of the implicit, and outdated, philosophy of science that encircle the term “falsifiability” while still maintaining a very reasonable standard that scientific work must have the potential of being resisted, and must be open to that resistance.

* For some interesting debates on the subject, see the NSF report on getting funded to do qualitative research, Howard Beckers’ excellent response “How to Find Out How to Do Qualitative Research”, both of which are discussed on Scatterplot here.
** A term I borrow from Luker’s excellent methods book, Salsa Dancing into the Social Sciences to describe (generally) qualitative and historical research that is more interpretive, meaning, context and history oriented.
*** A bit tangentially, but I wonder if this formulation might also help modern science studies scholars in their sometimes difficult quest to distinguish “tobacco science” from public health, or climate change skeptics from legitimate climatology (see, for example, Agnotology).