Readers of the blog likely know my fascination with quantification in all of its delightful and nefarious forms, including questionable rankings. A fellow UM Sociology student sent me a link to this excellent critique: Lies, damned lies, and ranking lists: The Top 100 Best NGOs. Interestingly, Algoso’s critique is not simply along the lines that the NGO rankings are arbitrary and their methodology a black box (though that is the case), but rather that NGOs, unlike (say) universities, are not sufficiently alike to be ranked at all:
So could we apply that metric/formula approach to NGOs? I don’t think so. As GJ points out, there’s no easy way to compare impacts across social sectors. At least universities are all doing basically the same thing (they educate students, conduct research, run athletic programs, etc.) and are structured in basically the same ways. But Wikimedia Foundation, Ashoka, TED, Search for Common Ground, and MSF? I could not think of a more diverse group of organizations in terms of missions, methods, or structures. How would you ever craft a set of metrics that would apply to all of these, let alone a formula that spits out a number to fairly rank them?
The problem is not simply that these particular rankings suck, but that the category “NGO” covers too heterogeneous a collection of objects to be worth ranking in the first place. Of course, as we know from the various branches of institutional theory, external forces like rankings often produce conformity when it is lacking to begin with. Whether or not NGOs will become more commensurable (is there an analog to earnings per share or return on equity?) remains to be seen, but it’s not impossible. For the moment though, rather than use the rankings to increase conformity, Algoso argues for dequantification – getting rid of the rankings entirely – on the grounds that this category is not sufficiently uniform to be worth ranking at all.*
If any other readers come across interesting examples of unusual or contested quantifications, please send them my way!
* This strategy is somewhat similar to the one my co-authors (Ellen Berrey and Fiona Rose Greenland) and I have written about in a new project on the dequantification of affirmative action at the University of Michigan. There, “race” was the contested, unsettled category whose lack of homogeneity imperiled a quantified system of admissions. Paper to be presented (hopefully) at ASA this year.