Abstract:

The a_d coefficient was developed to measure the within-group agreement of ratings. The underlying theory as well as the construction of the coefficient are explained. The a_d coefficient ranges from 0 to 1, regardless of the number of scale points, raters, or items. With some limitations the measure of the within-group agreement of different groups and groups from different studies is directly comparable. For statistical significance testing, the binomial distribution is introduced as a model of the ratings' random distribution given the true score of a group construct. This method enables a decision about essential agreement and not only about a significant difference from 0 or a chosen critical value. The a_d coefficient identifies a single true score within a group. It is not provided for multiple true score settings. The comparison of the a_d coefficient with other agreement indices shows that the new coefficient is in line with their outcomes, but does not result in infinite or inappropriate values.

Document Type:

DOI: 10.1348/000711009X465647