Educators strive to create "assessment cultures" in which they integrate evaluation into teaching and learning and match assessment methods with best instructional practice. But how do teachers and administrators discover and negotiate the values that underlie their evaluations? Bob Broad's 2003 volume, What We Really Value, introduced dynamic criteria mapping (DCM) as a method for eliciting locally-informed, context-sensitive criteria for writing assessments. The impact of DCM on assessment practice is beginning to emerge as more and more writing departments and programs adopt, adapt, or experiment with DCM approaches.

For the authors of Organic Writing Assessment, the DCM experience provided not only an authentic assessment of their own programs, but a nuanced language through which they can converse in the always vexing, potentially divisive realm of assessment theory and practice. Of equal interest are the adaptations these writers invented for Broad's original process, to make DCM even more responsive to local needs and exigencies.

Organic Writing Assessment represents an important step in the evolution of writing assessment in higher education. This volume documents the second generation of an assessment model that is regarded as scrupulously consistent with current theory; it shows DCM's flexibility, and presents an informed discussion of its limits and its potentials.

ORGANIC WRITING ASSESSMENT

Dynamic Criteria Mapping in ActionBy BOB BROAD LINDA ADLER-KASSNER BARRY ALFORD JANE DETWEILER HEIDI ESTREM SUSANMARIE HARRINGTON MAUREEN MCBRIDE ERIC STALIONS SCOTT WEEDEN

UTAH STATE UNIVERSITY PRESS

Copyright © 2009 Utah State University Press
All right reserved.
ISBN: 978-0-87421-730-8

Contents

1 Organic Matters: In Praise of Locally Grown Writing Assessment Bob Broad.......................................................................................................12 The Journey is the Destination: The Place of Assessment in an Activist Writing Program-Eastern Michigan University Linda Adler-Kassner and Heidi Estrem........................143 DCM as the Assessment Program: Mid Michigan Community College Barry Alford.....................................................................................................374 Designs on Assessment at UNR: University of Nevada, Reno Jane Detweiler and Maureen McBride....................................................................................525 Assessment Changes for the Long Haul: Dynamic Criteria Mapping at Indiana University Purdue University Indianapolis Susanmarie Harrington and Scott Weeden.....................756 Putting Placement on the Map: Bowling Green State University Eric Stalions.....................................................................................................1197 Bookend Bob Broad..............................................................................................................................................................154References........................................................................................................................................................................160Index.............................................................................................................................................................................164About the Authors.................................................................................................................................................................166

Chapter One

ORGANIC MATTERS In Praise of Locally Grown Writing Assessment Bob Broad

In his book Organic Matters (2001), farmer Henry Brockman criticizes the USDA definition of "organic" (grown without chemical pesticides or synthetic fertilizers) as dangerously weak. He points out that most commercially grown organic produce purchased at grocery stores lacks flavor and nutrition just as much as most commercially grown non-organic produce. Both these kinds of food are produced industrially with the goal of high yields, and with similar costs to the environment, to the flavor and nutritional value of the produce, and ultimately to consumers.

In other words, what Michael Pollan (2006) calls "industrial organic" agriculture enacts nearly (but not quite) as dramatic an abandonment of the human values and purposes of farming as does the dominant form of industrial agriculture, which also uses pesticides and petro-chemical fertilizers. Both forms of industrial farming ultimately fail to preserve or protect the land, and both fail to nourish the customer optimally; and both fail for the same reason: the pursuit of greater profits.

Brockman argues for (and practices) a tougher, more comprehensive standard of organic agriculture. This higher standard requires farmers to protect the ecosystem, select plant varieties for nutrition and taste rather than appearance and durability, and establish direct farmer-to-consumer connections through farmers' markets and Community Supported Agriculture co-operatives like those through which Brockman sells his produce. In my favorite passage from his slim book, he lays out the unanswered questions that prevented him from ever eating an organic tomato, imported from South America, brought to him by his sister one winter day a few years ago.

How could I [eat it]? I knew nothing about that tomato.... [Its] life history was a cipher to me. Who planted it? Who picked it? What kind of soil was it grown in? How was it fertilized? Irrigated? How many people had touched it on its long journey to my kitchen counter? How long had it sat in a box? Was the hangar, plane, truck, warehouse, cooler it sat in fumigated with noxious chemicals? How much fuel had been burned on its way from a field in Chile to my counter in Congerville [Illinois]? I had no idea what the answers to all these questions might be. This tomato was just too far removed from me and my life for me to eat. (Brockman, 1)

If the tomato comes from too far away for him to know its story, if the circumstances of its production and delivery to his home are, in Brockman's words (echoing Emmanuel Levinas's [1987] ethical philosophy) overwhelmingly "faceless," then Brockman would simply rather do without it altogether-even on a cold, tomato-barren December day.

Among educational leaders and reformers, the phrase "learning culture" is now commonplace (Shepard 2000). As educators, we nurture and grow our students' knowledge, skills, attitudes, and dispositions. We carefully tend the learning environment, and we provide our students with the best resources available to nourish their curiosity, understanding, and active participation in democratic citizenship. Among those of us interested particularly in assessment, we strive to create "assessment cultures" (Huot 2002) in which educators integrate their evaluations into teaching and learning (and vice versa) and match their assessment methods with best instructional practices (and vice versa).

Of course not everyone favors this "home gardening" approach to learning and evaluation. As this book goes to print, commercial testing corporations are eagerly inviting us to out-source writing (and other) assessments to their computerized systems (Ericsson and Haswell 2006). Note, for example, this recent postcard from the Educational Testing Service: "How long does it take you to evaluate an essay? Instantly ... using Criterion(tm) Online Writing Evaluation ..." (One's imagination flashes irresistibly to a hard, pale, joyless, imported tomato.) Or consider this subject line in a recent e-mail message from another evaluation corporation: "Faculty Unburdened: Assessment Made Simple in 5 Steps." Many of us feel troubled by such fast-food-style offers to make assessment faster and simpler by splitting it off from the rest of our work as educators. In such a climate, we need to recall and listen to other voices urging us to re-capture, re-coup, and harness organic, localized assessment to nourish productive teaching and learning.

In 1989, Egon G. Guba and Yvonna S. Lincoln published their book Fourth Generation Evaluation. Making good on the promise to their readers of a book "dramatically different from any other book about evaluation that you have ever read" (7), the authors issue a manifesto for a revolution in evaluation as a scholarly discipline and as an institutional practice. Though the paradigmatic and philosophical basis for-and performance of-this revolution is complex, one feature of it clearly marks it as a precursor to the evaluative approaches illustrated in the book you are now reading. Guba and Lincoln emphasize that the methods and findings of their evaluative system "are inextricably linked to the particular physical, psychological, social, and cultural contexts within which they are formed and to which they refer" (8). The authors go on to insist that...