Evaluations Work When Raters Want Them To

Chuck Schultz

Many evaluation systems produce good information for selection, supervision, or training needs assessment. But only when the raters use them as designed. A promotional evaluation can have great validity then fizzle within in two or three years. Or a supervisor may have good information from a needs assessment, but send employees to whatever training is available instead of what the employee needs.

Raters' attitudes toward the system may change. For example, when supervisors get flack from employees whose ratings are not sufficiently high for their purposes, the supervisors change their behavior. They may feel that other supervisors inflate their ratings. So they try to gauge how much to inflate their ratings to make them fair to their employees. Raters' second-guessing the system is a major cause of distortion and measurement error.

Of course, problems with ratings are well known. Rather than indicating employee differences, ratings may reflect the raters' biases or rating tendencies, such as leniency. Halo effect is often evident, as when a grumpy person gets lower than average ratings on everything. We try to develop objective systems, but raters can adulterate any of them.

Several high-level managers worked with me in developing a promotional evaluation based on a job analysis. We developed a five-point scale, with behavioral anchors for all five points. They loved it. They insisted on using a similar instrument for other job classes. It yielded good validity and the agency was happy with the results . . . the first year. There was a little grumbling the second year. The fourth year, tons of folks were discontented and the agency pitched it.

A colleague developed a forced-choice instrument, pairing empirically derived performance elements with neutral elements. They were balanced for desirability or frequency of use in describing promotional candidates. It worked great. It was hard to fake because it was difficult to figure out which alternative was keyed. When you have statements such as: "always responsible" and "respectful of co-workers" you need empirical data to decide which to key for a given job. Again, attitude with the instrument declined from enthusiastic to hostile. The raters couldn't even figure out how to cheat to make their employees' ratings fair.

It is possible for an uncontrolled, subjective rating system to work validly if a supervisor knows the performance of all the employees. Such a situation may exist in a clerical pool or a sports team. Subjective rating may work in such a situation, but I am not recommending it.

What I do recommend is: Use ratings only when raters want to use them as designed. Get the raters' ownership of the process. When you lose rapport, change the system.

Chuck Schultz may be reached at (360) 923-5340, 2941 A Firwood Loop SE, Olympia, WA 98501-4844.