CHAPTER 1

Overcoming Methodological Challenges

Questions about the relative merits of alternative research strategiespervade the social sciences. What counts as an adequate explanation forsocial phenomena? How can we evaluate competing explanations? Whatstandards should we apply when weighing evidence? How much andwhat types of evidence are convincing? Can social phenomena relatedto policy areas be studied scientifically? Some eminent scholars appearto agree on broad methodological goals or criteria (Brady and Collier2004; Gerring 2001; Lieberman 2005). Explanations should be generalyet precise, accurate, and well-specified. Evidence should be theoreticallyrelevant and should identify mechanisms linking explanations to outcomes.Abundant evidence, if theoretically relevant, is valued because itenhances confidence in findings.

Despite the apparent common ground underlying the work of manyscholars, methodological divides within the social sciences also run deep.As lamented by Mahoney and Goertz (2006) and E. Ostrom (2006), rivalcamps often cast aspersions on each other's work rather than engage inconstructive dialogue. The acrimony has several sources. The disagreementshave been provoked in part by battles over induction versus deduction,poor methodological practice by some scholars, and a lack ofsensitivity to diverse research goals. The stakes of the methodologicaldebate are increased by the intertwining of methodological choice withontological, normative, and theoretical positions, and with competitionfor professional status and resources (Moses and Knutsen 2007). Thesedynamics encourage intense and sometimes grossly unfair critiques.

The substantive focus of this book is on collective action and the commons.It is a field of research that utilizes multiple methods extensively,as well as being the one most familiar to the authors of this book. Webelieve that the discussion of the use of multiple methods in this researchfield, and the lessons we draw from our practical experiences, apply morebroadly to social science in general. Therefore, we start this first chapterwith a broader discussion on the methodological challenges in the socialsciences.

Examples of poor methodological practice pervade social science research.Often, scholars follow "the rule of the hammer" and apply asingle method indiscriminately, regardless of its suitability for a givenresearch project. Harmonization of research goals, theory, data, andmethod does not, however, guarantee sound practice. One can find qualitativestudies that overstate either the uniqueness or the generality ofparticular cases, fail to utilize relevant concepts and theories in the literature,or work with concepts that conflate multiple dimensions (Sartori1991; compare Goldthorpe 1997). Quantitative studies sometimes useinadequate data and do not always use appropriate diagnostic checksand technical fixes (Jackman 1985; Scruggs 2007; Shalev 2007). Formalmodels often work with unrealistic assumptions without addressing thegap between assumptions and reality (Bendor 1988; Green and Shapiro1994). No method is immune to poor applications.

Critics sometimes conflate methodological practice with the methoditself, arguing that examples of poor application discredit the method.A method need not be abandoned because it has been poorly utilized; itmakes more sense to encourage greater methodological awareness andbetter practices (Geddes 2003; Jackman 1985; King, Keohane, and Verba1994; Scruggs 2007). Others fail to appreciate that research goals arevaried and require diverse methods. More than three decades ago RobertClark (1977, 10; emphasis in original) strongly warned against relianceon a single method:

A first rule should be to beware of one researcher, one method, or oneinstrument. The point is not to prove that the hypothesis is correct, butto find out something. To rely on a single approach is to be shackled.

Indiscriminate application of a method makes little sense, but completerejection of a method because it is inappropriate in a particularsetting or for a particular purpose is not more sensible. It is importantfor social scientists to recognize that all methods generate results thatcontain some level of uncertainty. While multiple scientific goals andtrade-offs in achieving those goals are widely acknowledged (Coppedge1999; Gerring 2001), little consensus exists on the relative importanceof particular goals. Some scholars prioritize one or a few goals to suchan extent that they dismiss as unscientific research that prioritizes othergoals. For example, Goldthorpe (1997) includes generality as the mostimportant criterion in his definition of causal explanation, rather thanas one of several criteria (compare Gerring 2001). Consequently, he seesunique events and contingency as marking the limits of scientific inquiry.By this definition, analyses of such events are not scientific and cannotsupport causal inferences. Proponents of path-dependent explanations,analytic narratives, interpretive methods, and other approaches stronglydisagree (Bates et al. 1998; Bennett and Elman 2006; Rogowski 2004;R. Smith 2004). As in this example, and as discussed further below,methodological controversies often reflect competition between researchtraditions.

Fortunately, social scientists increasingly recognize trade-offs acrossmethods (Bates 2007; Brady and Collier 2004; Gerring 2001). King,Keohane, and Verba (1994), for example, point out that all methodologieshave limitations; scholars should be more aware of these limits andmore transparent about the limits as well as the solid contributions oftheir work. To overcome the limits of any one method, one needs todraw on multiple methods (Bates et al. 1998; Coppedge 1999; Granatoand Scioli 2004; Jackman 1985; King, Keohane, and Verba 1994; Laitin2003; Lieberman 2005; Scharpf 2000; Tarrow 2004). If social scientistshave shared standards, no single method fully addresses all standards.Methods offer different strengths and weaknesses. Rigorous researchthat combines complementary methods will be superior to research thatrelies on any single method (Gray et al. 2007).

The pragmatism and respect for diverse methodological traditions inthese reflections are welcome. Too often, however, the challenges involvedin using multiple methods are themselves overlooked. Proponentsof mixed methods justify their preferred combination in logical termsand illustrate the approach with a few examples. With some exceptions(Lieberman 2005; Scharpf 2000), this literature offers few specific practicalsuggestions.

Practical challenges can be formidable. Not all methods are equallyfeasible or even appropriate for all research topics (Bennett and Elman2006; Poteete and Ostrom 2008). Lieberman's (2005) nested analysis,for example, involves large-N analysis prior to any case study work.There are many important topics for which broadly comparative dataare scarce, difficult to access, or of dubious quality. Lieberman, however,does not address these challenges. Even if data availability is not aproblem, the value of a multimethod approach requires sufficient commandof multiple methods. Yet considerable investment is required togain competency in any methodology, and the benefits of methodologicalspecialization are substantial. While these challenges are sometimesacknowledged, few social scientists make practical suggestions to addressthem.

This book focuses on the practical challenges that influence methodologicalchoice. We are particularly concerned with research on topicsfor which data are scarce, difficult to collect, and not readily comparable.These conditions affect research on a wide variety of topics, includingthose concerned with informal institutions, subnational organizations,and nonelite populations. We focus on collective action for the managementof natural resources, an area of research in which all of theseconditions apply. For such topics, data for large-N analysis are neitheravailable nor readily accessible, and field research is unavoidable. Researchersoften need considerable contextual knowledge even to recognizethe phenomenon of interest. The need to conduct intensive fieldworklimits the potential for collecting enough data to support broadly comparativeanalysis.

We have become strongly aware of these challenges through our ownwork on collective action and natural resource management. We feel thatthe practical challenges of conducting rigorous social science researchon topics for which data are scarce, or difficult to access or to interpret,have not received adequate attention in discussions about social scienceresearch. We have seen the benefits of collaboration and the combinationof multiple methods in our research. We also have firsthand experienceof the challenges involved in such research, and we will discuss thesethroughout this book.

In this chapter, we introduce four themes that recur through the book:(1) the interlinking of methodological debates with theoretical development,(2) the advantages and limitations of multiple methods and collaborativeresearch, (3) practical constraints on methodological choices,and (4) the often problematic influence of career incentives on methodologicalpractice. In this book, we explicitly acknowledge the practicalchallenges that affect methodological choices, evaluate several strategiesfor addressing these challenges, and direct attention to the influence ofcareer incentives on methodological choices in social science research.We discuss a range of options for balancing competing methodologicaldemands under the inevitable conditions of limited resources, includinga variety of techniques that we feel have been underutilized in the socialsciences. We discuss the merits and limits of each method, as well as thepossibilities for and constraints on combining various methods. In ourdiscussion of constraints on methodological choice, we hope to stimulatea debate about professional incentives and other structural aspects ofacademia that influence how research is conducted.

This book is more about methodological practice than about methodologicalideals. We thus begin this chapter with a historical overview ofmethodological debates, highlighting interactions among methodologicalpractices, changing theoretical orientations, and competition for professionalstatus and resources. We then look more closely at issues surroundingresearch that uses multiple methods, an approach that has gained inacceptance in recent years. This leads to a discussion of constraints onmethodological choice, both practical and professional. We then explainhow our substantive focus—the study of collective action in natural resourcemanagement—helps us address our four thematic concerns. Thechapter concludes with an outline of the rest of the book.

Social Science Debates over the Superiority ofParticular Methods

The history of the social sciences can be recounted with reference to majormethodological shifts. An initial reliance on qualitative analysis gaveway dramatically to quantification in the early to mid-twentieth century.When this transformation began, quantification largely meant statisticalanalysis of large-N data sets of public opinion surveys. The last third ofthe twentieth century saw a surge in the use of formal models as well.Debates about the relative merits of qualitative, statistical, and formalmethods contributed to several developments in the late twentieth andearly twenty-first centuries: refinements of quantitative methods that attemptto better match social conditions; the rise of formal models; greaterappreciation for combining multiple methods; and the spread of post-positivistmethods such as discourse analysis.

The qualitative orientation of the early social sciences can be seen inthe emphasis on case studies and participant observation in sociology,ethnographic field-based research in anthropology, and descriptive andnormative analyses of formal legal arrangements. In the early decades ofthe twentieth century, many scholars embraced quantitative methods aspart of a drive to make the social sciences more scientific. Quantitativemethods began to gain currency across the social sciences in the 1920sand 1930s. The adoption of these methods accelerated at midcentury, asconveyed by references to the behavioral revolution.

The branches of the social sciences differed in their timing, pace, andpreferred forms of quantification. Nonetheless, the methodological shiftfrom qualitative to quantitative methods in the social sciences was dramatic.Psychology rapidly adopted experimental and statistical methods.Quantitative methods in economics encompassed formal models as wellas experiments and statistics. For sociology, research activities duringWorld War II marked the ascendance of survey research, experiments,and statistical forms of analysis (Platt 1986). Postwar political scienceshared the enthusiasm for survey research and statistical analysis, butformal modeling became widespread only in the 1980s and 1990s. Insociocultural anthropology, some interest was expressed in mathematicalmodels in the early postwar period, but multivariate statistical analysesremained relatively rare until the 1970s (Chibnik 1985).

The role of quantitative methods in the social sciences has always beencontentious. Current methodological debates echo those of a centuryago, even if framed in somewhat different terms. Scholars concernedwith methods have disagreed over (1) the goals of social research, (2)philosophical and theoretical issues, and (3) practical considerations,especially related to data quality. Methodological choices should bedriven by theoretical and ontological assumptions (Hall 2003), but theyalso reflect underlying values and beliefs (Mahoney and Goertz 2006)and practical considerations (Platt 1986). The ontological and normativedimensions of methodological choices are not widely recognized(Mahoney and Goertz 2006). As a result, social science debates aboutmethods involve frequent misunderstandings, with proponents of differentapproaches talking past each other (E. Ostrom 2006). Furthermore,because methodological discussions rarely acknowledge practical andprofessional considerations, they offer little guidance on how to addressthese constraints. In this section, we discuss controversies over the goalsof social research, and how philosophical and theoretical issues interactwith professional competition. We expand our treatment of practical andprofessional considerations in subsequent sections.

During the 1920s and 1930s, the social sciences became more institutionalizedin North America. The social sciences sought recognitionas sciences, and each discipline developed a more or less distinct professionalidentity (Guy 2003; Platt 1986). This process of institutionalizationinfluenced methodological debates. During the prewar period, disagreementsfocused on the goals of social research. Should sociological researchsupport social work to improve social conditions, seek subjectiveunderstanding of life experiences, or attempt to identify general patterns(Platt 1986)? Should the study of politics provide normative and practicalguidance for administrators or objective understanding of politicalphenomena (Guy 2003; Lasswell 1951)? As universities set up schools ofsocial work, public administration, and business administration alongsidedepartments of sociology, political science, and economics, differencesover goals were alleviated—but not really addressed—through theinstitutionalization of more focused programs of study.

Yet differences over the relative importance of theory and praxis cannotfully account for methodological debates. Scholars with commongoals disagree over methods, and scholars draw on the same methods topursue divergent goals. A lack of consensus on fundamental philosophicalissues contributes to disagreements over methods. What counts asscience? What model or models of causality and explanation make sensefor social phenomena? In particular, do models of science and explanationdeveloped in the natural, and especially the physical, sciences makesense for the social sciences?