Information Processing Approaches to Test Development and Construction as Evidence for Test Validity

by Dennis Doverspike

The tenets of personnel selection research have undergone many changes over the past thirty years (starting with a point coinciding with the enactment of the Civil Rights Act of 1964). This has been especially true of many of our ideas concerning the primacy of predictive studies as a validation strategy. We have seen rapid growth in our understanding of validation strategies and the introduction of a number of alternatives to traditional criterion-related validation included meta-analysis, validity generalization, content-oriented behavioral consistency approaches (Schmitt & Ostroff, 1986) and content/construct strategies (Binning & Barrett, 1989; Borman, Rosse & Abrahams, 1980; Landy, 1986). We have also seen an increased emphasis on the importance of the g factor in testing. One promising outcome of the rise of new validation strategies has been an increased concern with theory development. This has led many applied psychologists to look to recent advances in experimental psychology, especially artificial intelligence research, where progress has been made in developing new models of human information processing (IP).

The last three decades have also been marked by rapid technological change, with the introduction of the use of personal computers in the personnel field. In the area of testing, personal computers have already simplified the administration of tests and the collection of data. In addition, they also create the opportunity for the development of tests never before possible using a traditional paper-and-pencil test.

As we look forward to the year 2000, the computerized testing of IP abilities seems to offer unlimited growth potential and represents an attractive alternative to traditional paper-and-pencil tests of general ability. However, to this point in time, computerized IP tests have yet to live up to their full potential.

For the past 20 years, researchers at the University of Akron and elsewhere have been working on the problem of the development of IP based tests. The research has combined field and laboratory research. This approach is applicable to a variety of job types. Jobs which have been studied include secretarial, clerical, fire fighter, police officer, transport driver, process control, maintenance mechanic, and radar operator. The purpose of this article is to briefly describe this stream of research.

Definition

To start, we might ask what is an IP test. It is an ambiguous term and can refer to many different perspectives in cognitive and social psychology. In general, when we speak of an IP test we are speaking about a test which was based on an approach which viewed the human mind as in information processor that codes, stores and retrieves environmental inputs. A more specific definition would be that it is a general term for a class of both paper and pencil and computerized tests in which the subject's recognition, retention, reaction time, capacity or other responses are measured in response to verbal or perceptual stimuli. Another way of thinking of IP approaches is that they are cognitive consistency approaches. That is, in developing items on the test we try to create items which will create a press for cognitive operations which are isomorphic to those required by the job.

Development

When developing IP tests, there are a number of approaches which can be followed. While IP approaches appear to be particularly compatible with computerized testing, it is possible to develop paper-and-pencil IP tests. For paper-and-pencil testing, two basic developmental approaches have been used. The first is to adapt or select a test based on currently available IP tests on the IP constructs identified through a job analysis. The second approach would be to build or construct a test based on a construct/content strategy of test development.

For computerized IP tests several options are also available. One simple option is to convert a paper-and-pencil test into a computerized format. A second option is to turn to the experimental psychology literature, look for successful measures, and turn these measures into computerized IP tests. The third option would be to build or construct a test based on a construct/content strategy.

Regardless of the approach followed, the first step would be a job analysis. The IP tests construction process requires a much more detailed, intensive approach than might be normally required, although with experience this need not be a torturous experience. It does mean that we can not simply go out and ask job incumbents what IP's are required on their job. Rather the job analyst deduces the IP elements from an analysis similar to one used in protocol analysis. That is incumbents are asked to think/talk aloud about performance. We call such an analysis an IP job analysis (Barrett & Maurer, 1989; Arthur, Barrett & Doverspike, 1990).

If we look at the job analysis traditionally, we work at the behavioral level. The IP approach goes beyond this to the IP level. With the macro IP approach we are basically looking at decision points or what types of cognitive operations are being carried out in a general sense. Examples might be retrieving names from memory or a particular method of scanning a map. At the micro IP level we are attempting to extract even more information, for example how many bits of information have to be held in memory and for how long.

The second step is the identification of relevant constructs/content. This follows naturally from the job analysis and involves attempts to identify what are the critical IP tasks which can be measured. For any job, there may be many IP tasks. However, for some IP tasks individual differences will not be a factor. For others, we may not be capable of measuring the IP tasks or we cannot measure them in a reasonable fashion. This allows us to limit the set of IP elements to be measured. We then need to link IP elements to job behaviors.

The third step is the actual development of tests. Here, we can follow standard test development procedures including the development of appropriate instructions, decision on types of stimuli, number of stimuli, types of responses, numbers of responses, and types of measures (for example reaction time or number correct).

The fourth step is pilot testing. This may include experimental studies manipulating characteristics of the tests as well as traditional item analysis.

The fifth step is the development of the final version of the test. The sixth step is a continuation of the second step. This step involves documenting the links from job behaviors to Ips to test questions.

The seventh step involves studies of the validity of the tests as predictors of simulator performance. One of the unique aspects of our program has been the use of simulator performance as a criterion. Traditionally, we have lionized performance appraisal ratings as a criterion despite their well known weaknesses. Our argument is that a well constructed high fidelity simulation can serve as a more effective criterion than those often employed in criterion related study. The development of the simulator is also linked to the IP job analysis.

Conclusion

Computerized IP testing will continue to grow in importance as we move to the year 2000. While a number of problems still need to be solved, both computerized and IP testing of abilities is already a viable alternative to traditional testing. Unlike g measures, IP approaches can link testing and training, and serve as a foundation to both.

References

Arthur, W., Jr., (1987). The validity of information processing measures predicting accidents in simulated and real world contexts. Unpublished doctoral dissertation, The University of Akron, Department of Psychology, Akron, OH.

Barrett, G. V., & Maurer, T. J. (1989). The job analysis-predictor development process: Beyond g and generic constructs in employment aptitude testing. Unpublished manuscript, The University of Akron, Department of Psychology.

Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478-494.

Borman, W. C., Rosse, R. L., & Abrahams, N. M. (1980). An empirical construct validity approach to studying predictor-job performance links. Journal of Applied Psychology, 65, 662-671.

Landy, F. J. (1986). Stamp collecting versus science. American Psychologist, 41, 1183-1192.

Schmitt, N., & Ostroff, C. (1986). Operationalizing the behavioral consistency approach: Selection test development based on a content-oriented strategy. Personnel Psychology, 39, 91-108.

Dennis is a past chair of IPMAAC's University Liaison/Student Paper Competition and a frequent conference presenter. Dr. Doverspike may be reached at Psychology Department, University of Akron, Akron, Ohio 44325, phone: (216) 972-8372, fax (216) 762-5857, or e-mail (R1DD@VM1.CC.UAKRON.EDU)