EFL composition assessment: The relationships between teacher-rafer background characteristics and reader reliability

Abstract

ABSTRACT Title: EFL composition assessment: The relationships between teacher- rater background characteristics and reader reliability Author: Ali Sadkı Agazade Thesis Chairperson: Dr. Linda Laube, Bilkent University, MA TEFL Program Thesis Committee Members: Dr. Dan J. Tannacito, Dr. Ruth A. Yontz, Bilkent University, MA TEFL Program It is cannon knowledge among composition researchers that different composition readers judge the same compositions differently due to certain reader variables. This controversial issue leads to a further problem, which is best termed as the reader reliability and unreliability dichotomy. While it is assumed that some composition rater background characteristics influence reader reliability, few studies have investigated the rela tionships between rater background characteristics and reader reliability. This research study explored the relationships between composition teacher-rater background characteristics and reader reliability at the Eastern Mediterranean University Preparatory School in the Turkish Republic of Northern Cyprus. A background information questionnaire consisting of fourteen items was first developed, and then administered to 47 teacher-raters who had scored an EFL composition achievement test consisting of one prompt. The research participants scored a total number of 1191 student compositions. The arrangement of the scoring process was such that two teachers indepen dently scored a subset of about 30 compositions using a twenty-point analytic scale. The researcher was also provided with the composition assessment record prior to the administration of the questionnaire. The two sets of scores assigned to the compositions in each subset were correlated in order to identify the reliable and the unreliable readers within pairs. At this point, the dependent reliability variable was created so that it could be regressed with the independent background characteristics variables. One of the 14 independent variables - participation in reader group-training sessions before scoring Englishcomposition examination papers - was excluded from the study, and the analysis was done with 13 independent variables. This variable was excluded because all of the raters' responses were `yes` to this background characteristic. The analysis did not yield any statistically significant relation ships between the dependent variable and the independent variables group. Therefore, the alternative hypothesis stating that there are EFL composi tion teacher-rater background characteristics related to reader reliability was rejected, and the null hypothesis stating that there are no EFL composition teacher-rater background characteristics related to reader reliability had to be accepted. Only one rater out of the 47 raters was identified as unreliable. However, it is believed that factors other than background characteristics, such as fatigue, physical distractions, or subconscious rebellious behavior against the testing administrators affected his/her reading performance. This conclusion was supported by same collected data and insitu observa tions.