Technical Affairs Section

Mike Aamodt, Associate Editor

This month's column will answer two questions that were submitted by ACN readers. The first concerns how to determine adverse impact and the second the appropriateness of committee-based job analyses. Keep those questions coming!

Question: Should adverse impact be computed by the four-fifths rule or by the standard deviation approach? If it is the standard deviation approach, how does one compute this?

Answer: The two most common methods for determining adverse impact are the four-fifths rule and the standard deviation rule. From a legal or psychometric standpoint, it really doesn't matter which one is used. The four-fifths rule is typically used because it is better known and easier to compute.Let's use an example involving gender to demonstrate the use of the standard deviation rule. Suppose that 800 applicants apply for 200 openings as golf ball finders for the First Annual IPMAAC/U.S. Open Golf Tournament; 100 of the applicants are women and the other 700 are men. After thorough testing by the eminent scholar and golfer, Dr. Mark Nagy, we hire 19 females and 181 males. Do we have adverse impact? To answer this question using the standard deviation rule, we first compute the standard deviation by applying the following formula:

For our example, the numbers would be:

In our example, women make-up 12.5% (100/800) of the applicant pool. If we multiply this 12.5% by the number of openings (200), we would expect to hire 25 females. The logic behind the standard deviation rule is that the actual number of females hired should be within two standard deviations of what would be expected based on the number of females in the applicant pool. In our example, the fewest number of females that could be hired without demonstrating adverse impact would be (25 - (2 x 4.68)) = 25 - 9.36 = 15.64 or 16. Because the number of females we hired (19) exceeds the number represented by 2 standard deviations below what would be expected based on the number of female applicants (16), adverse impact is not demonstrated.

So, what would the results have been had we used the four-fifths rule? To answer this question we first determine the selection ratio for females and the selection ratio for males. In our example, the selection ratio for females would be 19/100 = .19. For males, the selection ratio would be 181/700 = .26. We then divide the female selection ratio (.19) by the male selection ratio (.26) which in our example would be .19/.26 = .73. Because .73 falls below .80 (four-fifths), adverse impact would have occurred.

Interestingly, the courts have ruled that if hiring one or two minorities (in this case females) would have placed our ratio above the magic 80% level, then adverse impact would not have legally occurred (Felix Waisome v. Port Authority of New York and New Jersey, Docket No. 88 Civ. 1234, January 29, 1991). In other words, the District Court ruled that the hiring process must be "scrutinized with an eye towards practical as well as statistical significance." In our example using the four-fifths rule, it might be argued that adverse impact did not occur because hiring two more females would have put us above the 80% level.

As you can see from the above example, the method used to determine adverse impact can make a difference. So, which one do we use? My advice would be to use the four-fifths rule as the primary method but also consider the standard deviation rule. If one shows adverse impact and the other not, it would be up to the courts to decide which of the two they will use. Let me also add that the courts have indicated that they don't have a preference for how adverse impact should be computed and retain the right to decide each case on its own merit rather than strict adherence to the Uniform Guidelines [Clady v. County of Los Angeles, 770 F. 2nd 1421 (9th Cir. 1985) and Watson v. Fort Worth Bank & Trust, 487 US 977, 108 S. Ct. 277,101, L.Ed.2d 827, on remand, 856 F.2nd 716 (5th Cir., 1988)].

Question: Is the committee-based method of job analysis an acceptable professional practice and would its use stand-up in court?

Short Answer: If subject matter experts are selected with care, the committee-based method is indeed an acceptable and common job analysis practice. As for its legal vulnerability, it should withstand legal challenge but who knows these days. Perhaps we can convince our ACN legal expert Jeff Feuquay to search for relevant case law.

Long Answer: Committee-based job analyses use a relatively small group of carefully selected subject matter experts (SMEs) who get together to generate a list of tasks performed on the job and the knowledge, skills, abilities, and other characteristics (KSAOs) needed to perform the tasks. In contrast, traditional field-based job analyses are conducted by interviewing and observing incumbents, having incumbents complete task and KSAO inventories, and then statistically summarizing the incumbents' task and KSAO ratings.

Obviously, there are tremendous time and cost saving advantages associated with the committee-based approach. In fact, some research indicates that by using a standard job analysis instrument and a committee-based approach, a job analysis can be completed in less than one hour! Because this seems too good to be true, we need to determine if the committee-based approach yields results similar to or better than the more time-consuming traditional approaches. To make this determination, we need to answer four questions:

1) Has any research compared committee-based and field job analyses?

2) Are there any "good" committee-based job analysis methods out there?

3) How many people need to be included in a job analysis? Can we get away with just a small number?

4) Does the composition of the committee matter? That is, can we put anyone on the committee or should it be composed in a particular way?

1) Has any research compared committee-based and field job analyses?

Yes, four studies that I am aware of. In the most recent study, Maurer and Tross (1996) compared the results of a nine-person job analysis committee with those of 38 employees in a field sample. The results of this comparison indicated agreement rates of about 85% for task ratings and 97% for KSA ratings. Similar results were found by Ash, Levine, Higbee, and Sistrunk (1982); O'Leary, Rheinstien, and McCauley (1990); and Tannenbaum and Wesley (1993). Takentogether, these four studies suggest that committee-based job analyses will yield similar results to field job analyses.

2) Are there any "good" committee-based job analysis methods out there?

Yes, there are several methods that have been successfully used for years. My personal favorite is the Ammerman Technique which was developed by H.L. Ammerman in 1965 and brought to the public's attention by David Robinson (the consultant, not the basketball player) in 1981. The basic steps to the Ammerman Technique are:

1) Convene a panel of experts that includes representatives from all levels of the organization.

2) Have the panel identify the objectives and standards that are to be met by the ideal incumbent.

3) Have the panel list the specific behaviors necessary for each objective or standard to be obtained.

4) Have the panel identify which of the behaviors from Step 3 are "critical" to reaching the objectives.

5) Choose tests that tap the KSAOs essential for the critical behaviors.

The Ammerman Technique works great with a wide variety of organizations. I have used it several times, encourage my students to use it, and have been happy with the results. On a side note, the Robinson (1981) article is, in my humble opinion, one of the best published examples of an interesting and innovative job analysis - if there is such a thing!

A second example of a good committee-based job analysis method is the Threshold Traits Analysis developed by Felix Lopez and his associates (Lopez, Rockmore, & Kesselman, 1980; Lopez, Kesselman, & Lopez, 1981). With Threshold Traits Analysis (TTA), a committee is formed and asked to rate the extent to which 33 "traits" are necessary for the successful performance of a job. Research indicates that TTA is short, reliable, and can correctly identify important traits. By the way, the Lopez et al. (1981) article describes a clever way to "validate" a job analysis instrument.

A third example of a committee-based job analysis is DACUM (Kosidlak, 1987). This approach was designed to identify essential competencies for health care professionals. Though no data were presented, Kosidlak (1987) reported successful results in using DACUM.

3) How many people need to be included in the job analysis?

This is a tricky question that I typically answer by telling my students to keep interviewing incumbents until you don't hear anything new. Anecdotally, this seems to be after the third or fourth incumbent for a particular job. However, because my anecdotes don't carry any weight with my wife, neighbors, or peers - my guess is that they won't with the readers of this column either. So, let's see what previous research can tell us.

Perhaps the first way to answer this question is to look at the interrater reliability typically found in job analyses. That is, if this reliability is high, incumbents tend to agree with one another; thus using ratings from 30 incumbents would not be any more meaningful than ratings from five incumbents. A look at current job analysis instruments suggests that interrater reliability is probably about .75 For example, interrater reliabilities for the Position Analysis Questionnaire range across studies from .45 to .96 with a mean of about .72. The interrater reliability for the Job Components Inventory is about .75 and about .84 for the Job Element Method.

The second way to answer this question is to look at research investigating or published advice regarding minimum job analysis samples sizes. Rouleau and Krain (1975) developed a table to estimate how many incumbents should be included in a job analysis. Their recommendation is that a committee-based approach should have 4-6 people and that one session should be fine for jobs having fewer than 30 incumbents and 2-3 sessions for jobs with higher numbers of incumbents. Gael (1988) suggests 6-10 SMEs are sufficient for what he calls an "SME conference."

In a very recent study, Beatty (1996) compared the results of job analysis samples of 10, 15, 20, and 212 incumbents in a federal law enforcement position. His results indicated that the job tasks and job requirements resulting from the use of 10 versus 212 incumbents were nearly identical. These results support and extend those found by Fletcher et al. (1993) and Pass and Robertson (1980) who found that job analysis samples of 10 and 20 yielded comparable results.

Mullins (1983) had 97 campus police officers at 13 universities generate critical incidents as part of a job analysis. The results indicted that no new incidents appeared after examining the incidents from the first three universities. Furthermore, after examining the incidents supplied by the first 19 incumbents, no new incidents or categories appeared.

Another study (Aamodt, Reardon, & Kimbrough, 1986) looked at the number of people needed to categorize critical incidents obtained during a job analysis. The results indicated that using as few as three incumbents to categorize incidents yielded results similar to as many as 100 incumbents.

4) Does the composition of the committee matter?

The idea behind involving mass quantities of incumbents in a job analysis is that people have a variety of views and by including only a small number of people in a job analysis, some of these views will be missed. So, if a committee consisting of a small number of people is to be used, research suggests that the committee members should represent those possible views.

SME Job Competence: Mullins and Kimbrough (1988) found that high performing SMEs generated different job analysis outcomes than low performing SMEs and Landy and Vasey (1991) found that experienced SMEs rated tasks differently than less experienced SMEs.

Race: Aamodt, Kimbrough, Keller, & Crawford (1982); Schmitt and Cohen (1989); Veres, Green, and Boyles (1991); and Landy and Vasey (1991) report small but significant differences in the ways in which white and black job incumbents viewed their jobs. For example, Landy and Vasey found that white police officers administered first aid more often and black police officers were more involved in sweeps and raids related to wide-spread narcotics use. Interestingly, Veres et al. found that job analysis ratings were related not only to the race of the incumbent, but the race of the incumbent's coworkers.

Gender: Landy and Vasey (1991) found possible differences in the ways males and females viewed their jobs. Because gender was confounded with experience, they were not able to draw any definite conclusions. Schmitt and Cohen (1989) found that male middle-level managers were more often involved in budgetary or finance related tasks than their female counterparts.

Educational Level: Landy and Vasey (1991) found high school educated officers to be less involved in court activities than their more educated counterparts.

Viewpoint: Given the popularity of 360 degree feedback, it should not come as a surprise that people with different perspectives on the job (e.g. incumbent, supervisor, customer) produce different job analysis results. For example, in an oldie but goodie, Wagner (1950) conducted a job analysis of dentists and found that patients generated more patient-dentist relationship critical incidents whereas dentists reported more technical proficiency incidents. Likewise, Andersson and Nilsson (1964) found differences in the critical incidents generated by grocery store managers, assistants, and customers. It is because of these differences that the Ammerman Technique panels are composed of people with differing views of the job.

Overall Conclusion: Based on the above information it would appear that reasonable job analysis results can be obtained by using a small (6-10 people), but carefully selected group/committee of SMEs. These SMEs would include members of appropriate demographic groups (e.g. gender, race) as well as representatives from relevant occupational groups (e.g. incumbents, supervisors, customers).

References

Aamodt, M. G., Kimbrough, W. W., Keller, R. J., & Crawford, K. J. (1982). Relationship between sex, race, and job performance level and the generation of critical incidents. Educational and Psychological Research, 2(4), 227-234.

Aamodt, M. G., Reardon, C., & Kimbrough, W. W. (1986). The critical incident technique revisited. Journal of Police and Criminal Psychology, 2(2), 48-59.

Andersson, B. E., & Nilsson, S. G. (1964). Studies in the reliability and validity of the critical incident technique. Journal of Applied Psychology, 48, 398-403.

Ash, R.A., Levine, E.L., Higbee,R.H., & Sistrunk,F. (1982).Comparisons of task ratings from subject matter experts versus job incumbents. Paper presented at the annual meeting of the Southeastern Psychological Association, New Orleans, LA.

Beatty, G. O. (1996). Job analysis sample size: How small is large enough? Poster presented at the annual meeting of the Society for Industrial and Organizational Psychology, San Diego, CA.

Fletcher, J., Friedman, L., McCarthy, P., McIntyre, C., O'Learly, B., & Rheinstein, J. (1993). Sample sizes required to attain stable job analysis inventory profiles. Poster presented at the 8th annual meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.

Gael, S. (1988). Subject matter expert conferences. In S. Gael (Ed.) The Job Analysis Handbook for Business, Industry, and Government (Vol I, page 434). New York: Wiley.

Kosidlak, J. G. (1987). DACUM: An alternative job analysis tool. Personnel, 64(3), 14-21.

Landy, F. J., & Vasey, J. (1991). Job analysis: The composition of SME samples. Personnel Psychology, 44, 27-50.

Lopez, F. M., Rockmore, B. W., & Kesselman, G. A. (1980). The development of an integrated career planning program at Gulf Power Company. Personnel Administrator, 25(10), 21-29.

Lopez, F. M., Kesselman, G. A., & Lopez, F. E. (1981). An empirical test of a trait-oriented job analysis technique. Personnel Psychology, 34, 479-502.

Maurer, T., & Tross,S. (1996). Committee vs. Field job analysis: Convergence, cautions, and a call. Poster presented at the annual conference of the Society for Industrial and Organizational Psychology, San Diego, CA.

Mullins, W. C. (1983). Job analysis outcomes as a function of group composition. Unpublished doctoral dissertation. University of Arkansas, Fayetteville.

Mullins, W. C., & Kimbrough, W. W. (1988). Group composition as a determinant of job analysis outcomes. Journal of Applied Psychology, 73(4), 657-664.

O'Leary,B.S., Rheinstein, J. &McCauley,D.E. (1990). Job analysis for test development: Can it be streamlined? Paper presented at the annual meeting of the American Psychological Association, Boston, MA.

Pass, J. J., & Robertson, D. W. (1980). Methods to evaluate scales and sample size for stable task inventory information (Report No. NPRDC TR 80-28). San Diego, CA: Naval Personnel Research and Development Center.

Robinson, D. D. (1981). Content-oriented personnel selection in a small business setting. Personnel Psychology, 34, 77-87.

Rouleau, E. J., & Krain, B. F. (1975). Using job analysis to design selection procedures. Public Personnel Management, 4, 300-304.

Schmitt, N., & Cohen, S. A. (1989). Internal analyses of task ratings by job incumbents. Journal of Applied Psychology, 74(1), 96-104.

Veres, J. G., Green, S. B., & Boyles, W. R. (1991). Racial differences on job analysis questionnaires: An empirical study. Public Personnel Management, 20(2), 135-144.

Wagner, R. F. (1950). A study of the critical requirements for dentists. University of Pittsburgh Bulletin, 46, 331-339.

Send your questions to Mike Aamodt: phone: (703) 831-5513, fax: (703) 831-6113, e-mail: maamodt@runet.edu.