When Rating Format Induces Different Rating Processes: The Effects of Descriptive and Evaluative Rating Modes on Discriminability and Accuracy

A third important purpose is a research function, notably when researchers are conducting criterion-related validation studies. Although our experiments do not deal with this kind of purpose, it is important to note that the concern in criterion-related validation studies is between-ratee discriminability and not within-discriminability. Thus, the results of the present research could also have implications for various research purposes for appraisal.

Murphy and Balzer (1986) obtained results that contradict this hypothesis. They showed that ratings had greater DEL with specific (behavior-based format) rather than global rating items (trait-based format). However, as their global and specific items were not comparable (global items were not derived from their specific items), we did not take these results into account to formulate our hypotheses.

The behaviors submitted to the pretests (N _EB = 60, N _DB = 60) were extracted from the most frequent behaviors in a large pool of EBs and DBs obtained in a pilot study in which 80 undergraduate students indicated the first EB (versus DB) that came to mind for the set of 40 traits initially selected.

These pilot tests were conducted with two groups of graduate students who were blind to the intended performance levels of each tape. The first group (N = 11) ranked the 7 managers according to their performance level. The second group (N = 14) ranked the 12 scales for each manager from the one on which the manager was the most competent to the one on which the manager was the least competent. For the results of the first test (ranking of the managers), although all differences were not significant, the managers were ranked as intended. For the results of the second test (ranking of the scales for each manager), although there was some variability in the ranking of the scales, in every case, a scale on which a manager’s performance was effective always received a higher ranking than a scale on which a manager’s performance was ineffective. It should be noted that although participants perceived differences in performance on the scales when the differences were great (i.e., differences over 4 ranks), they did not identify more subtle differences in performance (i.e., differences of 1 and 2 ranks). But this was not problematic. The important result was that the differences between the scales were sufficiently explicit so as to allow the participants to achieve discriminability within the targets.

The order of presentation of the 7 targets was excluded from all the analyses in studies 1 and 2 because no main or interaction effect implying it was significant.

Using restricted maximum likelihood estimation did not change the results in study 1 or in study 2.

We are grateful to an anonymous reviewer to have suggested this test.