earticle

영어

The current study aims to design The current study aims to design an analytic writing scale with reference to the TOEFL iBT® Scoring rubric from the Educational Testing Service (ETS) and to examine the reliability of the newly designed analytic rubric using English argumentative writing collected from 26 Korean undergraduate and graduate students. There are three main findings in this study: First, intra-rater reliability was examined among three raters showing a high correlation between the holistic and the analytic scorings. Second, inter-rater reliability was explored in terms of correlation coefficients, agreement rates (adjacent and perfect), Cohen’s Kappa coefficients, and Fleiss Kappa coefficients. In addition, the intraclass correlation coefficient (ICC) revealed a high correlation between the score on a new analytic rubric and the original holistic TOEFL score. Among the four rubric dimensions, ‘unity-progression-coherence’ and ‘language use’ showed relatively lower correlations than the other two dimensions (topic and task, organization and development). Also, the dimension that showed the highest correlation with the holistic ETS score is ‘organization and development.’ Moreover, ‘unity-progression-coherence’ contributed notably to the analytic assessment. A survey on raters’ perceptions of rubrics and scoring, and interviews suggested that they had challenges scoring based on the descriptors of two rubrics (analytic and holistic). This study has pedagogical implications for scoring writing, designing rubrics, and rater training.