Auszug. © Genehmigter Nachdruck. Alle Rechte vorbehalten.

MACHINE SCORING OF STUDENT ESSAYS

UTAH STATE UNIVERSITY PRESS

Contents

Introduction Patricia Freitag Ericsson and Richard H. Haswell....................................................................................11 Interested Complicities: The Dialectic of Computer-Assisted Writing Assessment Ken S. McAllister and Edward M. White..........................82 The Meaning of Meaning: Is a Paragraph More than an Equation? Patricia Freitag Ericsson.......................................................283 Can't Touch This: Reflections on the Servitude of Computers as Readers Chris M. Anson.........................................................384 Automatons and Automated Scoring: Drudges, Black Boxes, and Dei Ex Machina Richard H. Haswell.................................................575 Taking a Spin on the Intelligent Essay Assessor Tim McGee.....................................................................................796 ACCUPLACER's Essay-Scoring Technology: When Reliability Does Not Equal Validity Edmund Jones..................................................937 WritePlacer Plus in Place: An Exploratory Case Study Anne Herrington and Charles Moran........................................................1148 E-Write as a Means for Placement into Three Composition Courses: A Pilot Study Richard N. Matzen Jr. and Colleen Sorensen.....................1309 Computerized Writing Assessment: Community College Faculty Find Reasons to Say "Not Yet" William W. Ziegler...................................13810 Piloting the COMPASS E-Write Software at Jackson State Community College Teri T. Maddox.......................................................14711 The Role of the Writing Coordinator in a Culture of Placement by ACCUPLACER Gail S. Corso.....................................................15412 Always Already: Automated Essay Scoring and Grammar-Checkers in College Writing Courses Carl Whithaus.........................................16613 Automated Essay Grading in the Sociology Classroom: Finding Common Ground Edward Brent and Martha Townsend....................................17714 Automated Writing Instruction: Computer-Assisted or Computer-Driven Pedagogies? Beth Ann Rothermel............................................19915 Why Less Is Not More: What We Lose by Letting a Computer Score Writing Samples William Condon.................................................21116 More Work for Teacher? Possible Futures of Teaching Writing in the Age of Computerized Writing Assessment Bob Broad...........................22117 A Bibliography of Machine Scoring of Student Writing, 1962-2005 Richard H. Haswell............................................................234Glossary..........................................................................................................................................244Notes.............................................................................................................................................246References........................................................................................................................................251Index.............................................................................................................................................262

Chapter One

She knew how difficult creating something new had proved. And she certainly had learned the hard way that there were no easy shortcuts to success. In particular, she remembered with embarrassment how she had tried to crash through the gates of success with a little piece on a young author struggling to succeed, and she still squirmed when she remembered how Evaluator, the Agency of Culture's gateway computer, had responded to her first Submission with an extreme boredom and superior knowledge born of long experience, "Ah, yes, Ms. Austen, a story on a young author, another one. Let's see, that's the eighth today-one from North America, one from Europe, two from Asia, and the rest from Africa, where that seems a popular discovery of this month. Your ending, like your concentration on classroom action and late night discussion among would-be authors, makes this a clear example of Kunstlerroman type 4A.31. Record this number and check the library, which at the last network census has 4,245 examples, three of which are canonical, 103 Serious Fiction, and the remainder ephemera. (Landow 1992, 193-194)

This excerpt from George Landow's tongue-in-cheek short story about "Apprentice Author Austen" and her attempts to publish a story on the international computer network, thereby ensuring her promotion to "Author," suggests a frightful future for writing and its assessment. The notion that a computer can deliver aesthetic judgments based on quantifiable linguistic determinants is abhorrent to many contemporary writing teachers, who usually treasure such CPU-halting literary features as ambiguity, punning, metaphor, and veiled reference. But Landow's "Evaluator" may only be a few generations ahead of extant technologies like the Educational Testing Service's e-rater, and recent developments in the fields of linguistic theory, natural language processing, psychometrics, and software design have already made computers indispensable in the analysis, if not the assessment, of the written word. In this chapter, we approach the history of computer-assisted writing assessment using a broad perspective that takes into account the roles of computational and linguistics research, the entrepreneurialism that turns such research into branded commodities, the adoption and rejection of these technologies among teachers and administrators, and the reception of computer-assisted writing assessment by the students whose work these technologies process.

Such a broad treatment cannot hope to be comprehensive, of course. Fortunately, the field of computer-assisted writing assessment is sufficiently well established that there exist numerous retrospectives devoted to each of the roles noted above-research, marketing, adoption, and use-many of which are listed in the bibliography at the end of this book. Our purpose here in this first chapter of an entire volume dedicated to computer-assisted writing assessment is to offer readers a broad perspective on how computer-assisted writing assessment has reached the point it occupies today, a point at which the balance of funding is slowly shifting from the research side to the commercial side, and where there is-despite the protestations of many teachers and writers-an increasing acceptance of the idea that computers can prove useful in assessing writing. This objective cannot be reached by examining the disembodied parts of computer-assisted writing assessment's historical composition; instead, such assessment must be treated as an extended site of inquiry in which all its components are seen as articulated elements of a historical process. This complex process has evolved in particular ways and taken particular forms in the past half century due to a variety of social and economic relations that have elevated and devalued different interests along the way.

In the following sections we trace this web of relations and suggest that theoretically informed practice in particular circumstances-what we will be calling...