Über die Autorin bzw. den Autor

Diana C. Mutz is the Samuel A. Stouffer Professor of Political Science and Communication at the University of Pennsylvania.

Von der hinteren Coverseite

"Diana Mutz has written a marvelous introduction to population-based survey experiments. The book provides a masterful--and, witty!--consideration of the issues that differentiate these experiments from lab experiments, with all sorts of good pragmatic advice. She describes so many examples that I cannot imagine anyone reading it and not having at least one idea for a new experiment they might conduct themselves."--Jeremy Freese, Northwestern University

"A lucid discussion filled with accessible and wide-ranging examples. For political scientists, sociologists, and those in other allied fields, this book offers invaluable lessons from the cutting edge of social science."--Devah Pager, Princeton University

"With great clarity and insight--and dozens of fascinating examples--Mutz makes a compelling case for combining the strengths of large-scale surveys and tightly controlled experimental methods in tackling many of the most pressing issues in the social sciences. Students and professionals alike will find a wealth of practical advice in these pages about how and why to conduct population-based survey experiments."--Galen V. Bodenhausen, Northwestern University

"This accessible book is a valuable resource that explains concepts and applications equally well. It's both a practical primer for novice learners and a deep, definitive text for the new, rapidly expanding field of population-based experiments. Whether you skim for insights or dive into details, Mutz describes how what was once an impractical pipe dream is now a dream ripe for researchers to pluck for their next experiment."--Matthew Davis, University of Michigan

"The use of randomized experiments is the biggest change in the methodology of survey research in a generation. Mutz has been at the helm of this change. Time and again, Mutz dips into a treasure chest of exemplary experiments across the social sciences to illuminate issues of theory. All in all, this is the most intellectually engaging--and engagingly written--work I have read in years."--Paul Sniderman, Stanford University

"Diana Mutz has written an excellent first book-length treatment of this subject. Her writing style is informal--pleasantly so--and she is able to convey some relatively technical points in a clear manner that can be read by a wide audience. Population-Based Survey Experiments will be well received in the social sciences."--Rebecca B. Morton, New York University

Auszug. © Genehmigter Nachdruck. Alle Rechte vorbehalten.

Population-Based Survey Experiments

PRINCETON UNIVERSITY PRESS

Contents

List of Tables.....................................................................................................ixPreface............................................................................................................xiChapter One Population-Based Survey Experiments A Hybrid Methodology for the Social Sciences.....................1PART I TREATMENTS FOR POPULATION-BASED EXPERIMENTAL DESIGNS.......................................................23Chapter Two Treatments to Improve Measurement.....................................................................25Chapter Three Direct and Indirect Treatments......................................................................37Chapter Four Vignette Treatments..................................................................................54Chapter Five Treatments in the Context of Games...................................................................68PART II EXECUTION AND ANALYSIS....................................................................................81Chapter Six Execution of Population-Based Survey Experiments......................................................83Chapter Seven Analysis of Population-Based Survey Experiments.....................................................108PART III SITUATING POPULATION-BASED SURVEY EXPERIMENTS............................................................129Chapter Eight External Validity Reconsidered......................................................................131Chapter Nine More Than Just Another Method........................................................................155Bibliography.......................................................................................................161Index..............................................................................................................173

Chapter One

A HYBRID METHODOLOGY FOR THE SOCIAL SCIENCES

Approaches to scientific knowledge are a bit like rabid sports rivals; often they cannot say anything nice about their own team without simultaneously disparaging the other side. At some level, they know these intense rivalries would not exist if the other team were not a worthy contender, but the positive aspects of the other side are seldom acknowledged.

Likewise, empirical social scientists tend to develop expertise either in large-scale observational methods such as survey research, or in laboratory-based experimental approaches. They then spend the rest of their careers defending their choice of that particular approach in virtually everything they publish. Each time we submit a journal article, we rehearse all of the advantages of our own methodological choice, briefly mention its weaknesses, and make the case in no uncertain terms that what we have spent our time on is worthwhile. Go team! competition among methodological approaches is certainly implied, even if it is not explicitly stated. We do our best to defend our own ingroup by stressing the importance of internal validity if we have produced an experiment, or external validity if we have completed an observational study.

Fortunately, this caricature is gradually becoming less accurate, both in terms of its characterization of researchers—an increasing number of whom are trained in multiple methods—and in terms of how methodologists are focusing their attention. Although there are still many survey researchers working on improving their particular method, and many experimentalists focused on developing innovative experimental techniques, there are also methodologists paying specific attention to the problem of integrating results from experimental and observational studies. For the most part, these approaches involve applying complex statistical models to estimates of convenience sample-based experimental treatment effects in order to estimate what they might be in the population as a whole. The goal of population-based experiments is to address this problem through research design rather than analyses, combining the best aspects of both approaches, capitalizing on their strengths and eliminating many of their weaknesses. The purpose of this volume is to introduce scholars and students in the social sciences to the possibilities of this approach.

Defined in the most rudimentary terms, a population-based survey experiment is an experiment that is administered to a representative population sample. Another common term for this approach is simply "survey-experiment," but this abbreviated form can be misleading because it is not always clear what the term "survey" is meant to convey. The use of survey methods does not distinguish this approach from other combinations of survey and experimental methods. After all, many experiments already involve survey methods at least in administering pre-test and post-test questionnaires, but that is not what is meant here. Population-based survey experiments are not defined by their use of survey interview techniques— whether written or oral—nor by their location in a setting other than a laboratory. Instead, a population-based experiment uses survey sampling methodstoproduceacollectionofexperimentalsubjectsthatisrepresentative of the target population of interest for a particular theory, whether that population is a country, a state, an ethnic group, or some other subgroup. The population represented by the sample should be representative of the population to which the researcher intends to extend his or her findings.

In population-based survey experiments, experimental subjects are randomly assigned to conditions by the researcher, and treatments are administered as in any other experiment. But the participants are not generally required to show up in a laboratory in order to participate. Theoretically I suppose they could, but population-based experiments are infinitely more practical when the representative samples are not required to show up in a single location.

To clarify further, for purposes of this volume, when I use the term "experiment" in the context of population-based survey experiments, I am referring to studies in which the researcher controls the random assignment of participants to variations of the independent variable in order to observe their effects on a dependent variable. Importantly, the term "experiment" is often used far more broadly than this particular definition. For example, many classic "experiments" such as Galileo's observation of gravitational acceleration do not involve random assignment to conditions. And in the social sciences, Milgram's famous demonstration of obedience to authority initially lacked any second group or source of comparison, although he later added these to his design.

So while there are many important experiments that do not meet this definition, I exclude these types of studies from my definition of population-based survey experiments for two reasons. First, in order to be able to make clear statements about the contribution of population-based experiments to internal and external validity, I must limit discussion to experiments for which these two ends are indeed primary goals. Establishing causality and generalizing to a defined target population are not always the goals of research, but they are central to the majority of social scientific work. In addition, the type of experimentation I circumscribe is where population-based survey experiments have the most to offer. Other kinds of experimental studies undoubtedly could benefit from more diverse subject populations as well, but given that experiments that fall outside of this definition are focused on other purposes, this methodological development is less important to these types of studies. However, when scholars want to be certain that a given relationship involves cause and effect, and that their theory may be generalized beyond a narrow pool of subjects, then this is precisely the context in which population-based survey experiments can make their biggest contribution.

Strictly speaking, population-based survey experiments are more experiment than survey. By design, population-based experiments are experimental studies drawing on the power of random assignment to establish unbiased causal inferences. They are also administered to randomly selected, representative samples of the target population of interest, just as a survey would be. However, population-based experiments need not (and often have not) relied on nationally representative population samples. The population of interest might be members of a particular ethnic group, parents of children under the age of 8, those who watch television news, or some other group, but the key is that convenience samples are abandoned in favor of samples representing the target population of interest.

The advantage of population-based survey experiments is that theories can be tested on samples that are representative of the populations to which they are said to apply. The downside of this trade-off is that most researchers have little experience in administering experimental treatments outside of a laboratory setting, so new techniques and considerations come into play, as described at length in this volume.

Why now?

In one sense, population-based survey experiments are not new at all; simplified versions of them have been around at least since the early years of survey research in the United States. However, technological developments in survey research, combined with the development of innovative techniques in experimental design, have made highly complex and methodologically sophisticated population-based experiments increasingly accessible to social scientists across many disciplines. Unfortunately, aside from a few journal articles that have been contributed by early adopters of this technique, there has been no book to date addressing this topic in a comprehensive and accessible fashion.

Population-based experiments are neither fish nor fowl. As a result, the guidelines available in textbooks for each of these individual methods— for example, the considerations related to internal and external validity, the design advice, and so forth—do not address concerns specific to population-based experiments. The purpose of this volume is to fill this niche, and thus to encourage wider and more informed use of this technique across the social sciences.

Why is the population-based experimental approach emerging as a distinct methodological option only now? two technological innovations have brought about the emergence of this method. The first was the development of technology for computer-assisted telephone interviewing (CATI). Until the development of CATI, there were rigid constraints on experimental designs executed in the context of large population samples. The classic "split-ballot" experiment allowed for variation of a single facet, whereas current technologies allow for multiple variations of multiple factors. It has become unnecessary to produce many different versions of a paper questionnaire because the software simply does this for you, with the appropriate variation of the experimental stimulus automatically popping up on the interviewer's computer screen. This advance has allowed researchers to execute extremely complex experimental designs on large and diverse subject pools via telephone surveys.

In addition, the development of the internet has further expanded the possibilities for population-based experiments. Although internet-based interviewing of representative population samples is still in its infancy at this point, it is already possible to provide pictorial stimuli as well as video footage to random samples of respondents. The ability to exploit such dynamic data collection instruments has expanded the methodological repertoire and the inferential range of social scientists in many fields. Although population-based survey experiments were done by telephone or face to face long before internet-based interviewing emerged, the internet has greatly increased their potential.

The many advances in interviewing technology present social science with the potential to introduce some of its most important hypotheses to virtual laboratories scattered nationwide. Whether they are evaluating theoretical hypotheses, examining the robustness of laboratory findings, or testing empirical hypotheses of other varieties, scientists' abilities to experiment on large and diverse subject pools now enable them to address important social and behavioral phenomena with greater effectiveness and efficiency.

Who Uses Population-Based ExperimentS?

Population-based experiments can and have been used by social scientists in sociology, political science, psychology, economics, cognitive science, law, public health, communication, and public policy, to name just a few of the major fields that find this approach appealing. But the list does not end there. Population-based experiments have been utilized in more than twenty disciplines including psychiatry, anthropology, business, demography, African American studies, medicine, computer science, Middle Eastern studies, education, history, and even aviation studies. So long as the perceptions, behaviors, or attitudes of human beings are of interest, and the researcher's goal is to test a causal proposition of some kind, population-based survey experiments are likely to be valuable. But they are particularly so when the study is one that would benefit from combining the internal validity of experiments with the external validity of representative population samples.

My personal interest in population-based experiments stems in part from my experiences as an avid user of this method in my own research. In graduate school I was nominally trained in both survey and experimental methods, but these were conceived of as alternative rather than synthesizable approaches. The extent to which experiments were integrated with survey work was limited to tests of alternative question wording, the kind of study that was focused on minor methodological advances rather than substantively focused survey or experimental research. Given that I was not particularly interested in survey measurement issues, this did not seem like an exciting approach to me at the time. But just a few years later, I became aware of the potential this method offered for examining substantive research hypotheses and began incorporating it regularly into my own research.

Beginning in 200, Arthur (Skip) Lupia and I served as the original principal investigators involved in time-sharing Experiments for the Social Sciences (TESS), a large-scale infrastructure project supported by the national Science foundation which had as its mission to promote methodological innovation through the use of population-based survey experiments. Our inspiration for this program came from its intellectual forerunner, The Multi-investigator Study, which was spearheaded by Paul Sniderman of Stanford University. Paul originally gathered a group of scholars within the field of political science to share time on a single telephone survey. Each team of investigators was allotted a small amount of time on the survey, and all shared the core demographic questions. The theme that tied these studies together was methodological rather than substantive. Individually, the studies would make contributions to their respective fields and subfields. But collectively, by all using experimental designs, they would demonstrate novel ways to establish causality within the context of diverse population samples.

Skip Lupia and I were fortunate to be two of the young scholars who were invited to put experiments on the Multi-investigator Study. This platform gave us an opportunity to test our hypotheses in a new experimental context and advanced our research agendas substantially. This relatively simple, but powerful idea demonstrated the tremendous benefits of combining separately conceived and jointly implemented original studies. There were efficiencies of both time and money in this combined effort that meant that more researchers could engage in original data collection. TESS took this idea a step further by establishing an ongoing cross-disciplinary platform for research employing population-based survey experiments.

Our desire to provide this opportunity to social science writ large was the origin of the plan for time-sharing Experiments for the Social Sciences. Population-based survey experiments could be found here and there across the social sciences even before its inception in 200, but with TESS, we took the spirit and success of the Multi-investigator Studies and extended them to serve a greater number of researchers across a larger number of disciplines on an ongoing basis.

The advantages of TESS are straightforward from the perspective of users: it requires a minimum investment of investigators' time to propose a study, provides a quick turnaround time, and is free of charge as a result of generous support from the national Science foundation. Under these circumstances, few social scientists find reason to complain. In addition, there are broader benefits that accrue from the population-based experimental approach, which I outline in greater length later in this book.

As of 2009, TESS is under the able leadership of psychologist Penny Visser of the University of Chicago, and sociologist Jeremy Freese of northwestern University. It continues to offer graduate students and faculty from all over the world opportunities to run population-based experiments free of charge. Based on a simple streamlined online application process, proposals are reviewed within their respective disciplines, and once accepted they are placed on a data collection platform for execution on the population of interest. For details, interested researchers should visit the website, Experimentcentral.org, where the short application (maximum of five double-spaced pages!) and review process are explained.

Indeed, the bulk of the examples I draw on in this book come from TESS-sponsored studies. By adding greater flexibility to the instruments, and establishing a streamlined review process for proposals, we were able to serve an even greater number of scholars at a lower cost per experiment. Further, by expanding TESS outside of political science, we drew on the creativity and ingenuity of a much larger pool of scholars and a much broader range of research subjects. It is this insight that added so much to our own ideas about the breadth of potential applications for population-based experiments.

(Continues...)

Excerpted from Population-Based Survey Experimentsby Diana C. Mutz Copyright © 2011 by Princeton University Press. Excerpted by permission of PRINCETON UNIVERSITY PRESS. All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.