A GUIDE TO COLLEGE WRITING ASSESSMENT

UTAH STATE UNIVERSITY PRESS

Contents

Acknowledgments...........................................................................................................................................vii1 Introduction: Embracing the Power of Assessment.........................................................................................................12 Historicizing Writing Assessment........................................................................................................................143 Considering Theory......................................................................................................................................354 Attending to Context....................................................................................................................................595 Assessing Student Writers: Placement....................................................................................................................806 Assessing Student Writers: Proficiency..................................................................................................................947 Conducting Writing Program Assessments..................................................................................................................1098 Evaluating Writing Faculty and Instruction..............................................................................................................137Appendix A: Timeline: Contextualizing Key Events in the History of Writing Assessment.....................................................................157Appendix B: Writing Assessment: A Position Statement, the Conference on College Composition and Communication Committee on Assessment.....................161Appendix C: Sample Scoring Rubrics........................................................................................................................169Appendix D: Sample Classroom Observation Form.............................................................................................................174Appendix E: Sample Outcome-Based Student Survey...........................................................................................................177Appendix F: Sample Teaching Portfolio Table of Contents...................................................................................................180Appendix G: Sample Course Portfolio Directions............................................................................................................181Appendix H: Sample Course Portfolio Reading Guidelines....................................................................................................184Appendix I: Getting Started Guide for Program Assessment..................................................................................................186Appendix J: Sample Program Assessment Surveys.............................................................................................................188Appendix K: Sample Student Focus Group Outline............................................................................................................191Appendix L: Selective Annotated Bibliography of Additional Readings.......................................................................................193Glossary..................................................................................................................................................197References................................................................................................................................................206Index.....................................................................................................................................................216About the Authors.........................................................................................................................................219

Chapter One

This question, asked in an e-mail from a dean at a liberal arts college to the composition director, illustrates just how central writing and writing assessment have become to discussions about institutional assessment goals and practices that are occurring at colleges and universities across the country (and around the globe). When considered within a historical context, the contemporary embrace of writing as a means for evaluating learning outside of the composition classroom is not surprising. Writing, after all, has been linked to large-scale assessment ever since college entrance examinations evolved from oral tests of literacy abilities to written ones (Brereton 1995; Elliot 2005; Trachsel 1992) and is still a component of entrance evaluations at most institutions of higher education. Writing frequently plays a role in campus-wide assessments of individual student achievement as well, through rising-junior exams, graduation tests, and other competency certifications (Haswell 2001a; Murphy, Carlson, and Rooney 1993).

That a composition director would be included in discussions about institutional assessment is not surprising either, given that more and more program-level administrators are being asked to provide information for campus-wide self-studies and accreditation reviews. Colleges and universities are under such pressure these days to demonstrate the quality of their programs that it is rare for any administrator to be excluded from calls for assessment data of one kind or another. This is especially true for writing program administrators, who typically participate in cross-curricular general education initiatives by way of coordinating introductory composition courses and supporting the instructors who teach them.

What is, perhaps, most compelling about the e-mail query is the implicit message, conveyed by the second sentence, about the potential role of the composition director in the broad-based assessment this dean is beginning to imagine. The dean seems not to be ordering or cajoling the writing program administrator (WPA) to fall in line with an assessment regimen that has already been envisioned (as higher-ed administrative lore might encourage us to expect) but rather inviting the WPA to take an active part in designing and facilitating what promises to become a significant campus-wide initiative.

The proposition embedded within this e-mail is an important one indeed. As research shows, writing assessments do much more than simply allow administrators to demonstrate that their institutions, departments, and programs are successful; they have the power to influence curriculum and pedagogy, to categorize teachers and writers, and, ultimately, to define "good writing" (e.g., Hillocks 2002; O'Neill, Murphy, Huot, and Williamson 2005). In fact, specific writing assessments, especially those perceived to have high stakes for students and teachers, function as what Deborah Brandt (1998) calls "literacy sponsors" because they encourage and support the development of certain types of writing and writing abilities over others. In short, a department-level administrator who embraces assessment-especially the kind of assessment that extends beyond the boundaries of her specific program-is in a position not only to help set the agenda for campus-wide assessment initiatives, but to affect, even "transform," teaching and learning across the university community (Bean, Carrithers, and Earenfight 2005).

Unfortunately, while the particular WPA in this real-life scenario understood the positive aspects of involvement and was willing to help her dean think through how a college-wide writing initiative might be used, simultaneously, to evaluate learning across campus, many writing program administrators are not inclined to assume an active role in assessment-even when department chairs or deans show confidence in their doing so. A key reason for the reluctance is that while the negative aspects of program-level assessment are well known (and well publicized through listservs, conference presentations, and articles), the positive potential remains, to a large degree, unrealized-both by individual writing specialists and by composition and rhetoric, at large.

This guide is intended to help address what we see as both a serious problem and an overlooked opportunity: just as writing program administrators (and writing faculty, in general) are being asked to assume more responsibility for large-scale assessment, many are uninspired-or unprepared-to do so. Some resist the very idea of assessment efforts that seem externally motivated and, thus, ultimately unconcerned with improving student learning. Others struggle to justify the time and effort needed for an activity that often appears extraneous to the work they were hired to do (e.g., coordinate courses, supervise instructors, teach, conduct research, advise students, and so on). Still others understand the potential importance and relevance of large-scale assessments but have trouble making them work for their programs, faculty, and students.

We seek to meet the needs of a wide range of colleagues-those who direct (or help direct) writing programs and those who teach within them, those who are resistant to assessment generally and those whose prior experience with poorly conceived or inappropriate assessments has made them suspicious or cynical, and those who want to participate in-or even lead-large-scale assessment efforts but don't possess the knowledge to do so confidently or well. Our aim is not to minimize the challenges associated with assessment (there are many) but to help readers confront and contextualize these challenges so they will feel able to design and facilitate assessments that support the educational goals of their institutions and, in the process, enhance teaching and learning within their departments and programs. Because assessment is central to teaching and learning in general (Johnston 1989; Moss 1992; Shepard 2000) and to writing in particular (Huot 2002; White 1994), and because the stakes are so high for faculty and students, WPAs and their composition and rhetoric colleagues must find ways to help promote meaningful assessments and participate in the powerful acts of analyzing and using results. This guide's key contention is that creating the conditions that support meaningful assessment hinges on appreciating not only the range of available assessment practices but understanding the history and theories informing those practices as well as the critical components of our particular teaching contexts.

CONFRONTING THE CHALLENGES

As writing program administrators and faculty understand, far too often assessment initiatives are imposed from the top-down, rather than invited or encouraged. When assessment is imposed (or perceived to be imposed), its relevance may not be apparent. This is especially the case when people outside of a program (a dean, provost, or institutional effectiveness director) dictate the parameters of the assessment (e.g., the purpose(s), guiding question(s), and methods for data collection, analysis, reporting, and eventual use). An assessment that is not framed by questions important to the program administrators and faculty gathering the data and whose results, therefore, may not seem meaningful likely will be viewed as pointless busywork, completed simply to help others fill in the blanks of reports that, if they are read at all by decision-makers, will never be acted upon. Worse yet, if the purposes, audiences, and implications of externally initiated assessments are not made clear, program administrators and faculty may assume that results will be used in undesirable ways, for example, to exclude students, monitor faculty, and control curriculum, as has too often been the case at higher-ed institutions (e.g., Greenberg 1998; Gleason 2000; Agnew and McLaughlin 2001).

Negative feelings about assessment can be further exacerbated when program administrators are unfamiliar with possibilities for approaching large-scale assessment, as well as the key concepts, documented history, and recorded beliefs associated with various approaches. This unfamiliarity is reflected in multiple ways-through urgent postings on disciplinary list-servs asking for the "best way" to assess student work for course placement or curricular review, through assessment workshops in which program directors clamor for practical advice on how to confront administrative assessment mandates, and through the now-ubiquitous short articles in the Chronicle of Higher Education and elsewhere about tensions between various constituencies (e.g., faculty, university administrators, legislators) over the presumed "validity" and/or "reliability" of particular assessment methods.

Unfortunately, even the most informed responses to public pleas for assistance or reassurance do not magically solve the crises because, as assessment scholars know, good assessments are those that are designed locally, for the needs of specific institutions, faculty, and students. As a result, well-intentioned pleas often lead to poor assessments, which, in a circular way, can reinforce bad feelings about assessment generally. As Ed White (1994) and others have suggested, when writing program administrators are not knowledgeable or confident enough about assessment, they become vulnerable to individuals and agencies whose beliefs, goals, and agendas may not support writing curricula, pedagogy, and faculty, and may in fact conflict with what we define as best practices. Core disciplinary activities and values can be undermined by writing assessments that are at odds with what our scholarship supports. In short, when policymakers, university administrators, and testing companies-instead of knowledgeable WPAs and faculty members-make decisions about writing assessment, we risk losing the ability to define our own field as well as make decisions about our programs and students.

Unfamiliarity with approaches to large-scale writing assessment is understandable, given that many people charged with administering writing programs and facilitating program assessments do not have degrees in composition and rhetoric. A survey of composition placement practices conducted in the early 1990s indicated that while 97 percent of writing programs are administered by full-time faculty, only 14 percent of these administrators had a degree in composition and rhetoric or were pursuing scholarship in writing assessment (Huot 1994, 57-58). Similarly, research conducted later in the decade on employment prospects for composition and rhetoric specialists, indicated that there were more jobs in the field than specialists available to fill them (Stygall 2000). Given the relative stability of composition requirements over the past ten years and the concurrent reduction of tenure-track professorial lines nationwide, it is reasonable to expect that the number of non-specialists directing writing programs has increased (and will account for a large portion of the readership for this guide).

Yet, even a degree in composition and rhetoric does not guarantee familiarity with key aspects of writing program assessment. Though many writing administrators and faculty matriculated through composition and rhetoric programs that grounded them in composition theory and pedagogy, most are not familiar with the literature on large-scale assessment, nor did they take part in this type of assessment during graduate school. Sometimes the opportunities simply do not exist for gaining expertise and experience. Graduate courses that focus on assessment are relatively rare, for instance, and while teaching assistants may take part in large-scale assessments by reading placement portfolios or submitting sample first-year composition papers to the WPA, they aren't often asked to help design such assessments. When opportunities to learn about or participate more fully in assessment are provided, students do not always take advantage of them; despite evidence to the contrary, students do not believe they will ever need to know more than the assessment "basics" to succeed in their future academic roles.

As most experienced composition and rhetoric professionals know, however, many (if not most) positions in the field-whether tenure-line or not-include an administrative component, either on a permanent or rotating basis. In addition to highlighting general employment trends in the field, Gail Stygall (2000) notes that 33 percent of the composition and rhetoric positions advertised in 1998 included some form of administration-nearly a 10 percent increase since 1994 (386). Our more recent analysis of job ads suggests that the current percentage of positions requiring administration is more than 50 percent. Given that writing program administration of any kind necessarily involves assessment of curricula, student achievement, and/or faculty performance, it is reasonable to assume that a majority of composition and rhetoric specialists will not only end up administering programs but assessing them, whether or not they are sufficiently prepared to do so.

Without a background in large-scale assessment, WPAs and their composition and rhetoric colleagues may find concepts typically associated with such assessment strange and intimidating. Having developed their professional identities within the humanities, for the most part, they may cringe at references to "measuring" or "validating," which reflect a traditional social-science perspective. Though scholarship on writing assessment offers ways of negotiating liberal-arts values with those from the sciences, and though publication of such scholarship has increased significantly over recent years, it often goes unread. Until recently, much of the most useful literature was difficult to find, appearing in a seemingly scattered way in essay collections and journals focused on topics other than large-scale assessment. The more accessible literature, though not irrelevant, has often been of the "tool-box" type, focusing on methods used by a particular department or program with scant discussion of supporting research and theory. As a result, many writing specialists are confronted with terms, definitions, and interpretations imported from other disciplines with little knowledge about how they should be applied to situations that require evaluation of writing abilities, development, and instruction. Thus, many are left feeling unprepared to argue for the kinds of assessments that make sense on an intuitive level or, more likely, argue against those that appear inappropriate.

AN ILLUSTRATION

Cindy's early-career narrative provides a good illustration of how frustrating it can be to possess a basic understanding of current writing assessment practice, without having a real familiarity with assessment history and theory. Like many composition and rhetoric specialists, Cindy was hired right out of graduate school to direct a substantial writing program at a mid-sized university. The three years of practical administrative experience she obtained as a PhD student, along with the courses she took in composition theory and pedagogy, prepared her well to take on many of the challenges of her first position, including hiring, course scheduling, and faculty development. Unfortunately-and largely due to her own decisions (e.g., electing not to take her program's course in assessment)-her graduate-school apprenticeship did not fully equip her for what became one of the most important aspects of her position: designing, arguing for, and facilitating meaningful large-scale assessments.

During her first semester (fall 1998), Cindy was confronted with several assessment issues that needed to be addressed. Among these was a writing-course placement process that relied on a computerized multiple-choice exam taken by students during summer orientation. Many faculty and students complained about the exam, which seemed inappropriate in many ways. Among other problems, the exam rested on the assumption that students' ability to write well in college courses correlated with their ability to correctly answer questions about grammar and usage. However, because student placement was a university issue, affecting faculty, staff, and administrators outside of the English Department, Cindy and her colleagues could not just make changes unilaterally. In addition to speaking to other faculty within their department, they would need to consult staff in the testing office, the VP of student affairs, and other departments, such as mathematics, that relied on a similar placement test. They would need to convince others that the test was problematic and that there were viable alternatives.

(Continues...)

Excerpted from A GUIDE TO COLLEGE WRITING ASSESSMENTby PEGGY O'NEILL CINDY MOORE BRIAN HUOT Copyright © 2009 by Utah State University Press. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.