Technical Affairs

by Mike Aamodt, Associate Editor

This month's column contains an answer to an ACN reader's question, a commentary by your favorite Associate Editor, and yet another attempt at a little HR humor. If you have a technical question you want answered, please submit it via email [maamodt@runet.edu], voice [(540) 831-5513, or fax [(540) 831-6113}.

Question

Our police department recently started using a battery of personality tests recommended by a local clinical psychologist. The psychologist justified this battery by giving the tests to the top 10 officers and creating what she called a "profile of the model officer." Is this an acceptable method of job analysis?

Answer

No! No! No! This is not an acceptable method of either conducting a job analysis or a validity study. Unfortunately, I have seen this approach taken several times - usually when the "consulting expert" is a clinical psychologist or is not specifically trained in human resources or industrial/organizational psychology.

Why is this method not acceptable? The goal of a job analysis is to identify the tasks performed by an incumbent, the conditions under which those tasks are performed, and the knowledge, skills, abilities, and other characteristics (KSAO's) needed to perform the tasks under the conditions identified. A "profile analysis" does none of these three things. Instead, such an analysis merely identifies the typical person who is either attracted to a particular job or is hired by a particular organization. Let me use two examples from our data bank to illustrate the problem with using profiles (actually, it is more of a data safe deposit box than an entire data bank, but in this political season, embellishment is allowed).

A meta-analysis has clearly shown that successful police officers have elevated scores on the K, Pd, and Ma scales of the MMPI. Taken by itself, this "profile analysis" would suggest that we should hire applicants who match this profile: that is, defensive, asocial, and highly energetic. However, the meta-analysis indicates that unsuccessful police officers have this same profile, as do police applicants in general. Furthermore, the meta-analysis indicates that there is no significant relationship between scores on these scales and any aspect of police performance. In other words, there is a definite "police profile," but this profile is not at all related to performance.

A second example comes from a study we conducted on deputy sheriffs in a county jail. One of the interesting outcomes of this study was that the personality profile for these deputy sheriffs was markedly different from the typical "police profile." However, as was the case with the MMPI example in the previous paragraph, the profile for the high performing deputy sheriffs was identical to that of the low performing officers indicating a unique profile for people attracted to the field or hired by the Sheriff.

What, then, do these two examples demonstrate? They demonstrate the importance of conducting a professional job analysis and validation study and not relying on the quick and dirty "profile analysis" method. Now, profiles can be useful in understanding the skills and personalities of a particular field or organization. For example, we learned that compared to the general population, our correctional deputy sheriffs were a very detailed, rule oriented, critical group of folks. Though this knowledge probably will not help in selecting future employees, it might prove useful in predicting the success and acceptance of certain organizational development interventions.

Commentary

Recently, a student of mine was creating a "Styles of Handling Conflict" scale that he wanted to use in his own training workshops and eventually market. As part of this development process, he wanted to correlate his new test with several related tests (construct validity). He spent over $100 ordering a conflict styles test from a well-known publisher of organizational development instruments (you probably have their catalog).

After examining the test and the technical manual purchased at an additional cost, he realized that the test had no norms, no reliability information, and no validity information. Thinking this was a mistake, the student contacted the publisher and was surprised when he was told that no such information existed. The publisher added that people who purchased the test were asked to send back test scores so the publisher would eventually have some norms. Even worse, the author of the test told my student that "reliability and validity aren't important because the test was designed for training, not selection."

I told my student a test without reliability and validity is worthless - for any use - and suggested that he get his money back. After several weeks of arguing with the publisher and the author, he finally received a refund.

If this were an isolated incident, I would have shook my head and gone back to playing with my slide rule. Unfortunately, there seems to be a common perception in the training and organizational development fields that psychometric excellence is not needed if a test is only for training purposes. Such a perception is clearly wrong!

Suppose that a department head goes to a supervisory development workshop and, based on the results of a test, leaves with the perception that she is a caring extrovert. Leaders with this type of profile are typically trusted and well-liked by subordinates. Pleased with these results, the supervisor decides that she can skip future people-skills training. After all, the test said her people must love her! What she didn't realize is that even though the "test" had a pretty cover and used some fancy latent ink pens, there was no evidence to support the test results.

What if in actuality, she is a shy, cold person, hated by her employees? But hey, this is only training, the test does not need to be reliable or valid. What's the worse that can happen if we incorrectly measure an employee's personality, group orientation, stress level, time management ability, conflict resolution style, or any of 100 other topics that occur in training workshops? Eventually, it can lead to a number of problems including poor leadership, an unrealistic self-image, and lowered performance. Consequently, it makes no difference whether a test is used for training or some other reason. It must always be reliable and valid.

A test publisher is under an ethical obligation to ensure that its tests are psychometrically adequate - regardless of the test's intended use. The massive misuse of the Myers-Briggs Type Indicator is an excellent example of a poor quality test being used to make some very important decisions about people's lives. IPMAAC members should demand psychometric information on any tests purchased, voice their concerns about poor quality information, and then boycott any test publisher not meeting our standards.

Human Resource Humor: Some Terrible HR Nerd Jokes

Why did the personnel analyst cross the road?
So she could do a job analysis of the chicken.

How many affirmative action officers does it take to screw in a light bulb?
Only one, but he will need a variety of light bulbs.

Did you hear about the job applicant who stared into her freezer for two hours?
The orange juice can said "Concentrate."

What does a fire fighter drink after taking a promotion exam?
Whine.

What is a personnel analyst's favorite band?
Two Standard Errors.

What did the irritable applicant say to the personnel analyst?
Don't test my patience.

What did one personnel nerd say to the other before going skiing?
I'll intercept you at the slope!

Did you hear about the employee who insisted he had the right to wear short sleeves to work?
The 2nd amendment gave him the right to bare arms.