So, I attend a weekly team trivia event at a local pub here. It’s a fun event, around 40 teams of 2-6 players answer 6 rounds of questions on sheets of paper which are graded and then the top 3 teams get prize money based on the number of teams who paid to play. The questions are mostly pop culture (music and movies being two of the rounds each week) but range over everything from neuroscience to history to Harry Potter.
This week, one of the questions was a poorly written statistics question in a science category. I’ll repeat it here, shuddering all the while: “At a confidence interval of 95%, a value of .06 would be considered: Significant, trending towards significant, or not significant.” Now, if you’ve taken a few stats classes before, this question should make you want to rip out your hair and/or write an essay on the different philosophies of statistics.* As the resident quant on the team, it was up to me to decide how to answer the question.
There are, of course, a few problems with the question: first of all, the logic of hypothesis testing (significance, etc.) and logic of confidence intervals are very different, even if the underlying computations are very similar. So it doesn’t make much sense to talk about “significance” in terms of a confidence interval. But, being charitable to the trivia master (who was both drunk and is not a particularly math-y guy), I rewrote the question as: “At a confidence level of 95%, a p-value of .06 would be considered: Significant, trending towards significant, or not significant.” Now I have a dilemma: if option 2 is a thing, clearly it is the answer the question is going for, as .06 is about as close to significant (.05) as you can get without actually getting there. Of course, the way I was taught statistics, “trending towards significant” is not a thing, it’s something you might say when you don’t actually get a result, but not something that appears in a stats text book. So what do I do?
If I answer the way I think the question is looking for (“trending towards”), but it was actually a trick question, I will have no leg to stand on – I will have known the “correct” answer and guessed wrong. But if I answer the way I believe is less likely to be the one taken as correct, but the one I believe in, I will be able to argue about getting it wrong (perhaps to some effect, but probably not). And, most importantly, I will be able to account for my action in a satisfactory way that fits with my conception of myself (as “quant-y”, in this set of people anyway).
So I went with “not significant”. As predicted, the desired answer was “trending towards”. I argued, along with one of the econ grad students and a plastic surgeon, getting back one of three points the question was worth. Out team ended up tied for first and lost the tiebreaker.
Ok, that was unnecessarily long and probably boring. Take home points: people (at least, this one) act in ways such that they can account for their actions, even if this might seem to be the wrong move from an efficiency/instrumental standpoint (hat tip Goffman). Also, poorly worded stats questions at bar trivia nights make social science grad students angry.
* And indeed, this was the response of the team of economics grad students sitting across from us.