Quantifying the Qualitative: Increasing the Reliability of Subjective Language Assessments

This project explores how to increase the reliability of subjective language assessments. What are the major factors that affect reliability, and what can educators do to maximize it? Research in the field suggests four key factors that have the greatest effect on reliability. The first factor is the procedures used to administer and score the assessment. The more standardized the process is, the higher the reliability will be. The second factor is rater experience, both in the profession and with the assessment in question. Experience is positively correlated with reliability. Novice teachers who get training and feedback in a scoring rubric can become almost as reliable as veteran educators, so training cycles should be a component of any assessment program. The third factor that affects reliability is the type of task the students are asked to do. Tasks that integrate more than one language skill are more reliable than those that use only one skill. The fourth factor is student experience with the testing format. More experience is positively correlated with better achievement, so educators must make sure their students are familiar with the tasks they are asked to do.