Construct validity of multi-source performance ratings: An examination of the relationship of self-, supervisor-, and peer-ratings with cognitive and personality measures

Although more and more organizations prefer using multi-source performance ratings or 3601 feedback over traditional performance appraisals, researchers have been rather skeptical regarding the reliability and validity of such ratings. The present study examined the validity of self-, supervisor-, and peer-ratings of 195 employees in a Dutch public organization, using scores on an In-Basket exercise, an intelligence test, and a personality questionnaire as external criterion measures. Interrater agreement ranged from .28 to .38. Variance in the ratings was explained by both method and content factors. Support for the external construct validity was rather weak. Supervisor-ratings were not found to be superior to self- and peer-ratings in predicting the scores on the external measures.