A Model for a proficiency/final achivement test for use at Erciyes University preparatory school

Abstract

ABSTRACT Title: A model for a proficiency/ final achievement test for use at Erciyes University Preparatory School Author: Faruk Balkaya Thesis Chairperson: Dr. Dr. Arlene Clachar Bilkent University, MA TEFL Program Thesis Committee Members: Dr. Phyllis L. Lim, Ms. Patricia J. Brenner, Bilkent University MA TEFL Program The goal of this study was to develop and pilot a model of a test based on course objectives that could be used for both a proficiency test and an achievement test for Erciyes University Preparatory School (EUPS) and that could be demonstrated to have reasonable reliability and validity. Only general English and reading skills were included in this pilot study. This newly developed model test, the Erciyes University Proficiency/Final Achievement Test (EUPFAT, or PAT, for short), consisted of 64 open-ended items such as short-answer, sentence completion, interrogatives, and rational cloze as recommended by a number of researchers (e.g., Heaton, 1988; Hughes, 1989; Hill & Parry, 1992). No multiple-choice items were included as it has been suggested that they can produce negative backwash (Hughes, 1989). Twenty- two items testing general English skills and 42 items testing reading comprehension were included. There were 35 intermediate-level English as a Foreign Language students attending the prep school who volunteered to pilot the PAT. Of these 35 subjects, 30 also took the English as a Second Language Achievementm Test (ESLAT) (1984), which was one of two criteria for estimating validity of the PAT. Teachers' evaluations of the 35 subjects who took the PAT were also used, as the second criterion. Following piloting, the PAT was scored independently by two scorers using an answer key prepared by the researcher. Inter-rater reliability was.99. The PAT was then evaluated for reliability and validity. Item analysis was also performed to identify items that should be replaced or rewritten for future administration of the tests. For internal consistency, the split-half reliability estimate of Pearson Product-Moment Correlation adjusted for length by Spearman-Brown Prophecy Formula, the Guttman split-half reliability estimate, the K-R 20, and the K-R 21 reliability formulas were used. The reliability coefficients estimated for internal consistency using these different split-half methods ranged from.87 to.96. The descriptive statistics of the PAT are as follows: N = 35, Mean = 29.86, Variance = 110.89, Standard Deviation = 10.53, Sum of Item Variance = 11.93. To determine the correlation between the PAT and the ESLAT, and between the PAT and the teacher evaluations, Pearson Product Moment Correlation (PPMC) was used. PPMC between the PAT and the ESLAT is.61, df = 28, £ <.0004, and the correlation between the PAT and the teacher evaluation of subjects is.74, df - 33, £ =.0000.IV Item analysis of the PAT has demonstrated that if the 19 of 64 items which are lying outside the acceptable range for item difficulty and discriminability, are eliminated from the test, the rest of the test items can be used as part of a proficiency/final achievement test. Because the total number of subjects in this study was not very high (N = 35), generalizing the results to other EFL situations should be avoided. However, the results of this study should be taken into consideration while developing a new test, or evaluating existing tests by those who are interested or involved in language testing.