Questionnaires

Case Study: Judy Ng Commentary: J.D. Brown

CASE STUDY

Introduction

This chapter reports on aspects of an in-depth study I conducted at a private university in Malaysia which sought to investigate lecturers' beliefs about the value of feedback on their students' written work and the extent to which the lecturers' practices converged with their self-reported beliefs. The study adopted various procedures for data collection – including interviews (see Chapter 4), think aloud (see Chapter 6), stimulated recall (see Chapter 7) and focus groups (see Chapter 3) – but the specific focus of the chapter is the design and administration of the questionnaires issued to the lecturers.

The value of teachers' feedback on students' writing has always been controversial, some researchers advocating various forms of feedback (e.g. Ferris, 1995, 2009), while others (e.g. Truscott, 1996, 2007) deny its usefulness. It has long been recognised that what teachers do in practice is largely influenced by their beliefs and values (Clarke & Peterson, 1986) and one of the most vital aspects of this issue is the study of the contextual factors that shape teachers' beliefs (Borg, 2006; Goldstein, 2005). While there have been wide-ranging surveys of aspects of teacher beliefs in some Asian educational contexts (Littlewood, 2007; Nunan, 2003), there is a lack of in-depth studies, and this is particularly true of tertiary education in Malaysia. In his comprehensive overview of studies of language teachers' beliefs, Borg (2006) pointed to the need for more studies in areas such as second-language writing. He also emphasised the need to examine the extent to which teachers' reported beliefs are actually put into practice.

Methodological Focus

Brown (2001: 6) defines questionnaires as 'any written instruments that present respondents with a series of questions or statements to which they are to react either by writing out their answers or selecting from among existing answers' and he points out that questionnaires can elicit individuals' reactions to, perceptions of and opinions of an issue or issues. Other functions of questionnaires include: obtaining demographic information about respondents (Creswell, 2005); scoping the respondents (Cohen et al., 2007); and eliciting and measuring abstract, cognitive processes, individual preferences and values (Brown, 2001; Dörnyei & Taguchi, 2010; Wagner, 2010).

The issue of sampling is essential in a questionnaire study (Brown, 2001; Creswell, 2005; Dörnyei & Taguchi, 2010). The major difficulty with sampling is in the accurate representation of the target population (Dörnyei & Taguchi, 2010; Gillham, 2000). It is usually very difficult and expensive to obtain a probability sample (Dörnyei & Taguchi, 2010: 60) that reflects the actual population. However, individual researchers can get reliable results using various non-probability sampling strategies. Some of the procedures for choosing the samples include random sampling, stratified random sampling, convenience sampling, quota sampling and snowball sampling (in which the researcher identifies key respondents who meet the criteria for the research and with their help is able to contact and recruit other participants).

A questionnaire can be administered in a number of ways, such as via post, email or online (Creswell, 2005; Dörnyei & Taguchi, 2010). However, there is a risk of getting a low return rate (Brown, 2001). There are, though, two approaches that give high return rates: one is one-to-one administration, whereby the questionnaires are personally delivered to the respondents and collected either there and then or later; another way is through group administration, when all the respondents are gathered together in the same place.

Most questionnaires contain both closed- and open-ended items. The former require respondents to choose from predetermined responses to various (Likert-type) scales, to select multiple-choice items or to rank order various alternatives (Brown, 2009; Dörnyei & Taguchi, 2010; Gillham, 2000). Although the range of possible responses is restricted, such close-ended items enable a straightforward analysis of the data. Several authors (Brown, 2001, 2009; Cohen et al., 2007; Dörnyei & Taguchi, 2010; Gillham, 2000) have suggested the use of more open-ended questions for smaller numbers of respondents and those who are doing qualitative research. Open-ended items typically include short-answer questions to elicit information which might otherwise have been overlooked; they can also serve to cross-check the validity of responses (Brown, 2009; Dörnyei & Taguchi, 2010).

It is necessary to pilot the questionnaire to ensure that the questions are not ambiguous and to check the feasibility of the procedures (Gillham, 2000; Wilson & McLean, 1994).

Once the questionnaires are completed and collected, the data need to be analysed. The type of analysis depends on the nature of the research and the number of respondents. If the number of respondents and/or items is large, Dörnyei and Taguchi (2010) recommend the use of a spreadsheet program such as Microsoft Excel or the Statistical Package for the Social Sciences (SPSS). Open-ended questions are more difficult to analyse due to the potential wide range of response types. In order to solve this problem, Brown (2001) suggests grouping similar responses together with the aid of a software program; for instance, recent versions of NVivo (Bazeley, 2007) can assist with the collation, management, coding and triangulation of data from multiple sources, including questionnaires.

Two recently published studies were particularly relevant to my research; Lee (2003, 2009) used questionnaires to examine whether teachers' beliefs converged with their reported practices. The relevance of these two studies lies less in their findings (which, will, however, be briefly discussed below) than in their use of questionnaires to provide baseline date for subsequent phases of data collection; Lee's (2009) study followed up an initial survey of 206 teachers by telephone interviews with a convenience sample of 19 participants. Lee (2009) went on to indicate 10 mismatches between teachers' beliefs and written feedback practice. In my view, such findings, although plausible, cannot be accurately inferred from the participants'...