Examining Raters’ Cognitive Processing in IELTS Speaking Assessment

L2 Raters as human beings are wide open to biased and inconsistent patterns of judgment which are commonly known as rater effects or measurement errors. This study was a qualitative research attempted to explore such inaccurate rating discrepancies in IELTS raters’ cognitive representations of the English speaking assessment criteria. To pursue this aim and to route Han’s cognitive processing modelled map (2016), ten expert and experienced IELTS raters were purposefully recruited for attending verbal protocols in four phases (Han, 2016) to encode the rater participants’ cognitive representations of (a) conceptualizing IELTS speaking rubric, (b) assessing the recorded response to Task Two in IELTS interview, (c) assigning scores and (d) revising/finalizing the assigned scores. Data collection was followed by transcribing, segmenting, encoding and analyzing the contents of the recorded verbal protocol reports. After content analysis of the recorded verbal protocols, the four categories of (1) Grammatical range and accuracy (2) Fluency and coherence (3) themes. The themes distributions in IELTS Speaking rubric were calculated as Category 1:7.64%, Category 2: 34.22%, Category 3: 43.85%, and Category 4: 14.29% by QSR NVivo 8.The researcher’s interpretations of the research findings were incorporated into the research questions. In verbal protocols: Phase I, the themes distributions in rater participants’ verbal protocol reports were the lowest in Category 4 (28.47%) and the highest in Category 2 (46.53%) to demonstrate the raters’ coral attention to natural flow of the candidates’ speech more than other categories in their cognitive representation of the IELTS speaking rubric. In verbal protocol: Phase II, the themes distributions in rater participants’ verbal protocol reports were again the lowest in Category 4 (24.40%) and the highest in Category 1 (63.12%)to support the raters’ given priority to the accuracy and grammatical soundness of the candidates’ speech production in generating their cognitive representation of the response. In verbal protocol: Phase III, the themes distributions in rater participants’ verbal protocol reports were again the lowest in Category 1 (37.50%) and the highest in Category 2 (66.07%) to prove the rater participants’ preference for the candidates’ naturalness of speech and expanding the topic properly in assigning their scores. Finally, in verbal protocol: Phase IV the themes distributions in rater participants’ verbal protocol reports were the lowest in Category 4 (32.10%) and the highest in Category 3 (53.25%) to reveal the rater participants’ redirected attention to the candidates’ abundant vocabulary knowledge and appropriacy of word choices. The implications of this research for training IELTS raters, coaching IELTS candidates and inspiring researchers in L2 assessment were enlisted as the researcher’s concluding remarks

کلیدواژه ها:

Assessment ، Cognition ، Cognitive processing ، Rater ، Scoring ، Scoring rubric ، Speaking