Arab Educational Information Network

Although the classical theory has been widely used to verify the psychometric characteristics of scales, this theory cannot give a general reliability coefficient with a single analysis. This study used the generalizability theory, which helps to determine the relative importance of various sources of error and yields optimal generalizability and dependability coefficients, to assess the validity and reliability of students' evaluation of instruction. The study data was obtained by selecting a random sample of 300 classes at Taibah University. The data was analyzed using generalizability theory. The object of measurement was the class and the error variance facets of items and students were determined along with the interaction effects. A generalizability study was conducted to calculate generalizability and dependability coefficients and to determine the validity and reliability of the questionnaire scores. Next, a decision study was performed using the components of variance identified by the generalizability study to find a method to minimize the impact of error variance. The results showed that the coefficients of generalizability and dependability for the course evaluation scores were suitable and that their values were affected more strongly and positively by increasing the number of evaluation items than by increasing the number of evaluating students. Using this method, we obtained a multi-faceted interpretation of error variance that can inform decision making regarding the circumstances under which accurate and generalizable assessments can be made. This interpretation resulted in more comprehensive and clearer psychometric evidence than can be obtained from validity and reliability coefficients that are determined with the classical theory. (Published abstract)