Statistical measurement of interobserver agreement [: Analysis of agreements and disagreements between observers]

When a patient is examined by a physician, it is desirable that the findings (diagnosis, symptoms) do not change when she is examined by a different physician. The patient is not inspired with confidence if physicians seriously differ in opinion, just because they cannot all be righL Of course, the foregoing does not only apply to physicians, but applies to judging observers in general, especially in situations where it is impossible to establish the truth in an objective way. In medical and non-medical scientific research associations between different characteristics (personality aspects, housing conditions. diagnosis, medical history, degree of recovery) may be blurred by errors in the assessment of these characteristics. Thereby important associations may be overlooked. Associations that are discovered may be hard to interpret because different studies may result in different quantitative conclusions. The specification of disagreement may be the first step to a better agreement among observers: Which disagreements do frequently occur and which observers often have different opinions? This may lead to a further standardization of the way of judging and to regular training on the basis of difficult cases; this second step, however, is not considered in this book. The value of judgements, in daily practice and in scientific research, may greatly increase by substantially improved interobserver agreement. For the sake of completeness, however. it must be mentioned that even perfect interobserver agreement does not guarantee that judgements have any value, Therefore, when simplifying a judgement system, one must avoid to lose important information. In the present book experiments are considered where subjects (e.g. patients) are placed in categories by at least two observers (e.g. physicians). The categories are determined in advance and are the same for all observers. A subject is assigned by an observer to one category. It is not necessary that all subjects are judged by the same observers, and also the number of observers per subject may vary. With respect to a particular subject. there is perfect agreement among the observers if they all use the same category. The possible disagreements between two observers are not equally serious if the categories are ordered