Readers of the blog likely know my fascination with quantification in all of its delightful and nefarious forms, including questionable rankings. A fellow UM Sociology student sent me a link to this excellent critique: Lies, damned lies, and ranking lists: The Top 100 Best NGOs. Interestingly, Algoso’s critique is not simply along the lines that the NGO rankings are arbitrary and their methodology a black box (though that is the case), but rather that NGOs, unlike (say) universities, are not sufficiently alike to be ranked at all:
So could we apply that metric/formula approach to NGOs? I don’t think so. As GJ points out, there’s no easy way to compare impacts across social sectors. At least universities are all doing basically the same thing (they educate students, conduct research, run athletic programs, etc.) and are structured in basically the same ways. But Wikimedia Foundation, Ashoka, TED, Search for Common Ground, and MSF? I could not think of a more diverse group of organizations in terms of missions, methods, or structures. How would you ever craft a set of metrics that would apply to all of these, let alone a formula that spits out a number to fairly rank them?
The problem is not simply that these particular rankings suck, but that the category “NGO” covers too heterogeneous a collection of objects to be worth ranking in the first place. Of course, as we know from the various branches of institutional theory, external forces like rankings often produce conformity when it is lacking to begin with. Whether or not NGOs will become more commensurable (is there an analog to earnings per share or return on equity?) remains to be seen, but it’s not impossible. For the moment though, rather than use the rankings to increase conformity, Algoso argues for dequantification – getting rid of the rankings entirely – on the grounds that this category is not sufficiently uniform to be worth ranking at all.*
If any other readers come across interesting examples of unusual or contested quantifications, please send them my way!
* This strategy is somewhat similar to the one my co-authors (Ellen Berrey and Fiona Rose Greenland) and I have written about in a new project on the dequantification of affirmative action at the University of Michigan. There, “race” was the contested, unsettled category whose lack of homogeneity imperiled a quantified system of admissions. Paper to be presented (hopefully) at ASA this year.
A (Budding) Sociologist's Commonplace Book 
Dave Algoso
/ January 26, 2012Hi Dan, I’m glad you liked the post! I’m really interested in the comment you make about how “external forces like rankings often produce conformity when it is lacking to begin with”. I don’t think this will happen with NGOs because I don’t think the rankings will carry *that* much weight. But maybe I’m wrong. Any chance you could point me to some resources on the topic?
Dan Hirschman
/ January 26, 2012Dave,
Gladly! The most relevant citations are Wendy Espeland and Michael Sauder’s work on “rankings and reactivity” and “the discipline of rankings,” though those are precisely about the USNWR university rankings (so already a more homogenous group, I would agree).
More generically, there’s a huge literature in organizational theory on “institutional isomorphism,” or the process by which organizations come to look and act more similarly due to external forces (but not simply competitive ones). Some introductory cites would be Scott’s Institutions and Organizations or DiMaggio and Powell’s classic article from 1983 and the follow-up edited volume. Off the top of my head, I can’t think of an article that’s specifically framed around how a very heterogeneous group is made much more homogeneous through rankings in particular. Perhaps some of my other org theory-minded readers can supply their favorite empirical stories in this vein?
cemferreira
/ January 29, 2012I find it particularly interesting that the very term NGO is a negative definition: the term Non-Governmental Organization describes only what these orgs are NOT, not what they are. Maybe they could be ranked in terms of impact, if impact could be defined – impact in terms of what, with whom.
An example from my own work is biodiversity – there is no such thing as biodiversity, as the definition is multi-layered; it is a byword for “nature”, covering everything from the genetic to the ecosystem level. And despite that, a lot of effort and investment is being deployed to measure it.