CHAPTER 1

Overcoming Methodological Challenges

Questions about the relative merits of alternative research strategiespervade the social sciences. What counts as an adequate explanation forsocial phenomena? How can we evaluate competing explanations? Whatstandards should we apply when weighing evidence? How much andwhat types of evidence are convincing? Can social phenomena relatedto policy areas be studied scientifically? Some eminent scholars appearto agree on broad methodological goals or criteria (Brady and Collier2004; Gerring 2001; Lieberman 2005). Explanations should be generalyet precise, accurate, and well-specified. Evidence should be theoreticallyrelevant and should identify mechanisms linking explanations to outcomes.Abundant evidence, if theoretically relevant, is valued because itenhances confidence in findings.

Despite the apparent common ground underlying the work of manyscholars, methodological divides within the social sciences also run deep.As lamented by Mahoney and Goertz (2006) and E. Ostrom (2006), rivalcamps often cast aspersions on each other's work rather than engage inconstructive dialogue. The acrimony has several sources. The disagreementshave been provoked in part by battles over induction versus deduction,poor methodological practice by some scholars, and a lack ofsensitivity to diverse research goals. The stakes of the methodologicaldebate are increased by the intertwining of methodological choice withontological, normative, and theoretical positions, and with competitionfor professional status and resources (Moses and Knutsen 2007). Thesedynamics encourage intense and sometimes grossly unfair critiques.

The substantive focus of this book is on collective action and the commons.It is a field of research that utilizes multiple methods extensively,as well as being the one most familiar to the authors of this book. Webelieve that the discussion of the use of multiple methods in this researchfield, and the lessons we draw from our practical experiences, apply morebroadly to social science in general. Therefore, we start this first chapterwith a broader discussion on the methodological challenges in the socialsciences.

Examples of poor methodological practice pervade social science research.Often, scholars follow "the rule of the hammer" and apply asingle method indiscriminately, regardless of its suitability for a givenresearch project. Harmonization of research goals, theory, data, andmethod does not, however, guarantee sound practice. One can find qualitativestudies that overstate either the uniqueness or the generality ofparticular cases, fail to utilize relevant concepts and theories in the literature,or work with concepts that conflate multiple dimensions (Sartori1991; compare Goldthorpe 1997). Quantitative studies sometimes useinadequate data and do not always use appropriate diagnostic checksand technical fixes (Jackman 1985; Scruggs 2007; Shalev 2007). Formalmodels often work with unrealistic assumptions without addressing thegap between assumptions and reality (Bendor 1988; Green and Shapiro1994). No method is immune to poor applications.

Critics sometimes conflate methodological practice with the methoditself, arguing that examples of poor application discredit the method.A method need not be abandoned because it has been poorly utilized; itmakes more sense to encourage greater methodological awareness andbetter practices (Geddes 2003; Jackman 1985; King, Keohane, and Verba1994; Scruggs 2007). Others fail to appreciate that research goals arevaried and require diverse methods. More than three decades ago RobertClark (1977, 10; emphasis in original) strongly warned against relianceon a single method:

A first rule should be to beware of one researcher, one method, or oneinstrument. The point is not to prove that the hypothesis is correct, butto find out something. To rely on a single approach is to be shackled.

Indiscriminate application of a method makes little sense, but completerejection of a method because it is inappropriate in a particularsetting or for a particular purpose is not more sensible. It is importantfor social scientists to recognize that all methods generate results thatcontain some level of uncertainty. While multiple scientific goals andtrade-offs in achieving those goals are widely acknowledged (Coppedge1999; Gerring 2001), little consensus exists on the relative importanceof particular goals. Some scholars prioritize one or a few goals to suchan extent that they dismiss as unscientific research that prioritizes othergoals. For example, Goldthorpe (1997) includes generality as the mostimportant criterion in his definition of causal explanation, rather thanas one of several criteria (compare Gerring 2001). Consequently, he seesunique events and contingency as marking the limits of scientific inquiry.By this definition, analyses of such events are not scientific and cannotsupport causal inferences. Proponents of path-dependent explanations,analytic narratives,...