Über die Autorin bzw. den Autor

Diana C. Mutz is the Samuel A. Stouffer Professor of Political Science and Communication at the University of Pennsylvania. She is a member of the National Academy of Sciences.

Von der hinteren Coverseite

"Diana Mutz has written a marvelous introduction to population-based survey experiments. The book provides a masterful--and, witty!--consideration of the issues that differentiate these experiments from lab experiments, with all sorts of good pragmatic advice. She describes so many examples that I cannot imagine anyone reading it and not having at least one idea for a new experiment they might conduct themselves."--Jeremy Freese, Northwestern University

"A lucid discussion filled with accessible and wide-ranging examples. For political scientists, sociologists, and those in other allied fields, this book offers invaluable lessons from the cutting edge of social science."--Devah Pager, Princeton University

"With great clarity and insight--and dozens of fascinating examples--Mutz makes a compelling case for combining the strengths of large-scale surveys and tightly controlled experimental methods in tackling many of the most pressing issues in the social sciences. Students and professionals alike will find a wealth of practical advice in these pages about how and why to conduct population-based survey experiments."--Galen V. Bodenhausen, Northwestern University

"This accessible book is a valuable resource that explains concepts and applications equally well. It's both a practical primer for novice learners and a deep, definitive text for the new, rapidly expanding field of population-based experiments. Whether you skim for insights or dive into details, Mutz describes how what was once an impractical pipe dream is now a dream ripe for researchers to pluck for their next experiment."--Matthew Davis, University of Michigan

"The use of randomized experiments is the biggest change in the methodology of survey research in a generation. Mutz has been at the helm of this change. Time and again, Mutz dips into a treasure chest of exemplary experiments across the social sciences to illuminate issues of theory. All in all, this is the most intellectually engaging--and engagingly written--work I have read in years."--Paul Sniderman, Stanford University

"Diana Mutz has written an excellent first book-length treatment of this subject. Her writing style is informal--pleasantly so--and she is able to convey some relatively technical points in a clear manner that can be read by a wide audience. Population-Based Survey Experiments will be well received in the social sciences."--Rebecca B. Morton, New York University

Auszug. © Genehmigter Nachdruck. Alle Rechte vorbehalten.

Population-Based Survey Experiments

PRINCETON UNIVERSITY PRESS

Contents

List of Tables.....................................................................................................ixPreface............................................................................................................xiChapter One Population-Based Survey Experiments A Hybrid Methodology for the Social Sciences.....................1PART I TREATMENTS FOR POPULATION-BASED EXPERIMENTAL DESIGNS.......................................................23Chapter Two Treatments to Improve Measurement.....................................................................25Chapter Three Direct and Indirect Treatments......................................................................37Chapter Four Vignette Treatments..................................................................................54Chapter Five Treatments in the Context of Games...................................................................68PART II EXECUTION AND ANALYSIS....................................................................................81Chapter Six Execution of Population-Based Survey Experiments......................................................83Chapter Seven Analysis of Population-Based Survey Experiments.....................................................108PART III SITUATING POPULATION-BASED SURVEY EXPERIMENTS............................................................129Chapter Eight External Validity Reconsidered......................................................................131Chapter Nine More Than Just Another Method........................................................................155Bibliography.......................................................................................................161Index..............................................................................................................173

Chapter One

A HYBRID METHODOLOGY FOR THE SOCIAL SCIENCES

Approaches to scientific knowledge are a bit like rabid sports rivals; often they cannot say anything nice about their own team without simultaneously disparaging the other side. At some level, they know these intense rivalries would not exist if the other team were not a worthy contender, but the positive aspects of the other side are seldom acknowledged.

Likewise, empirical social scientists tend to develop expertise either in large-scale observational methods such as survey research, or in laboratory-based experimental approaches. They then spend the rest of their careers defending their choice of that particular approach in virtually everything they publish. Each time we submit a journal article, we rehearse all of the advantages of our own methodological choice, briefly mention its weaknesses, and make the case in no uncertain terms that what we have spent our time on is worthwhile. Go team! competition among methodological approaches is certainly implied, even if it is not explicitly stated. We do our best to defend our own ingroup by stressing the importance of internal validity if we have produced an experiment, or external validity if we have completed an observational study.

Fortunately, this caricature is gradually becoming less accurate, both in terms of its characterization of researchers—an increasing number of whom are trained in multiple methods—and in terms of how methodologists are focusing their attention. Although there are still many survey researchers working on improving their particular method, and many experimentalists focused on developing innovative experimental techniques, there are also methodologists paying specific attention to the problem of integrating results from experimental and observational studies. For the most part, these approaches involve applying complex statistical models to estimates of convenience sample-based experimental treatment effects in order to estimate what they might be in the population as a whole. The goal of population-based experiments is to address this problem through research design rather than analyses, combining the best aspects of both approaches, capitalizing on their strengths and eliminating many of their weaknesses. The purpose of this volume is to introduce scholars and students in the social sciences to the possibilities of this approach.

Defined in the most rudimentary terms, a population-based survey experiment is an experiment that is administered to a representative population sample. Another common term for this approach is simply "survey-experiment," but this abbreviated form can be misleading because it is not always clear what the term "survey" is meant to convey. The use of survey methods does not distinguish this approach from other combinations of survey and experimental methods. After all, many experiments already involve survey methods at least in administering pre-test and post-test questionnaires, but that is not what is meant here. Population-based survey experiments are not defined by their use of survey interview techniques— whether written or oral—nor by their location in a setting other than a laboratory. Instead, a population-based experiment uses survey sampling methodstoproduceacollectionofexperimentalsubjectsthatisrepresentative of the target population of interest for a particular theory, whether that population is a country, a state, an ethnic group, or some other subgroup. The population represented by the sample should be representative of the population to which the researcher intends to extend his or her findings.

In population-based survey experiments, experimental subjects are randomly assigned to conditions by the researcher, and treatments are administered as in any other experiment. But the participants are not generally required to show up in a laboratory in order to participate. Theoretically I suppose they could, but population-based experiments are infinitely more practical when the representative samples are not required to show up in a single location.

To clarify further, for purposes of this volume, when I use the term "experiment" in the context of population-based survey experiments, I am referring to studies in which the researcher controls the random assignment of participants to variations of the independent variable in order to observe their effects on a dependent variable. Importantly, the term "experiment" is often used far more broadly than this particular definition. For example, many classic "experiments" such as Galileo's observation of gravitational acceleration do not involve random assignment to conditions. And in the social sciences, Milgram's famous demonstration of obedience to authority initially lacked any second group or source of comparison, although he later added these to his design.

So while there are many important experiments that do not meet this definition, I exclude these types of studies from my definition of population-based survey experiments for two reasons. First, in order to be able to make clear statements about the contribution of population-based experiments to internal and external validity, I must limit discussion to experiments for which these two ends are indeed primary goals. Establishing causality and generalizing to a defined target population are not always the goals of research, but they are central to the majority of social scientific work. In addition, the type of...