Practice Exchange

Ilene Gast, Associate Editor

This column highlights innovative assessment programs of interest to assessment professionals. If you are conducting a project that would interest the ACN readers, or if you know someone who is, please let me hear from you. I can be reached by phone at (202) 305-0590, fax at (202) 305-3664, or electronic mail at ilene.f.gast@justice.usdoj.gov.

Competency Based Behavioral Interview (CBBI): Variations on a Theme

by Charles Brooks, Jeff Maile, and Jason Fehr

Often, our customers ask, "What interview questions can I use to get good employees?" We tell them that "The questions you ask in an employment interview are not important." After they recover, we qualify that statement. We emphasize that the questions are irrelevant unless you know how to score the answers to the questions. Thus, the scales you use are of paramount importance.

At the Georgia Merit System, we develop and apply interviews that identify high performers with a proven 80% accuracy rate. We know how to ask relevant questions and we know how to score the answers. We have discovered some strategies that work.

1. Link the interview and the scales to high performance.

Begin by talking to the high performers. Identify the competencies associated with high performance and develop a competency model. Each competency must include a scale illustrated with behavioral examples at each point in the scale, especially the high performance points. You will get these examples when you talk to the people who engage in high performance. Average performers, by definition, don't exhibit them, so we don't ask them for examples of high performance. Ask people who, at the very least, recognize high performance when they see it. High performing incumbents, supervisors, and/or managers are the only valid sources of the data we need. If you average performance data across random samples (i.e., if you do not stratify your sample by the performance criteria) you will create a model of average performance.

2. In the interview, ask for examples of past behavior rather than hypothetical speculations.

When you are interviewing a candidate, elicit examples of actions, thoughts, feelings, strategies, etc. - Ask them what they actually did. Then, ask follow-up questions to clarify the candidate's behavior. Your follow-up questions will depend on what the candidate tells you. With one, you might need to ask, "Who exactly was 'we'? What role did you play?" With another you might ask, "Tell me the specific steps you took." Your aim is to find out exactly how the candidate operates in important job situations.

3. Cross-validate the model and/or the interview.

Develop the competency scales and interview questions with a sample of appropriate experts. Try it out with a different sample that you have stratified by performance levels. If the interview correctly differentiates high performers from average and low performers, it's valid.

4. Train the interviewers in practice, not theory.

We use videotaped scenarios to train our interviewer/raters and to calibrate their ratings. The scenarios are based on exemplars obtained from the criterion samples. This training enables us make sure that our interviewer/raters can uniformly recognize the behavioral indicators and apply the scales correctly and consistently. That automatically corrects for rater errors. Then, if the interviewers elicit appropriate data in the interview, through competency-based questions and practiced use of follow-up questions, the interview will predict quite well.

This system has been implemented in numerous projects at the Georgia Merit System, most notably with Child Support and the Georgia Bureau of Investigation.

Child Support Enforcement Agent (CSEA)

In this project, incumbents were asked to give several examples of situations that either went well or did not go well and to describe, in their own words, what they did, thought, felt and said in each situation. This process, called Behavioral Event Interviewing (BEI), was recorded on audiotape and later transcribed.

In all, we conducted 33 interviews; 18 were used to develop the profile of high performance and the remaining 15 were used to validate this profile. The initial sample had 12 high performers and 6 low performers, while the validation sample had 10 high performers and 5 low performers.

We scored the first set of BEIs using generic competency scales developed by Spencer and Spencer (1993). Our analysts identified seven competencies that were either differentiating or threshold competencies. On differentiating competencies, high and low performers differ significantly in their level of performance. On threshold competencies, the level of performance is consistent across the two groups. These seven competencies were included in the profile of high performance. We excluded several other competencies from our final model, because they showed no consistent differences or similarities between high and average performers. We refined the model by using the behaviors obtained during the interviews to modify the generic, competency-based rating scales.

We then evaluated the validation sample of the BEI transcripts using the empirically modified scales. We scored and ranked the incumbents based on the sum of their scale scores. We also ranked the incumbents based on prior performance data. The competency-based rank ordering yielded an 80% accuracy rate. Only one average performer was misidentified as a high performer and two high performers were categorized as average. We then developed focused interview questions targeting each competency in the model (i.e., the CBBI), and interviewed nine new incumbents for whom we had prior performance ratings and objective collections information. We again achieved an 80% accuracy rate. Only one high performer was misidentified as average.

Georgia Bureau of Investigation (GBI) Special Agent

We used a similar methodology to develop a CBBI for GBI Special Agent. The GBI, however, did not have the resources for an empirical study of known performers. Job-related competencies had been identified previously during a traditional, content validity job analysis, which was conducted for the purpose of developing a written multiple-choice test. We used a competency list developed by the Office of Personnel Management to facilitate this process. With professional assistance from the Merit System, a subject matter expert (SME) panel composed of supervisors and high-performing incumbents, prepared and selected behaviorally-focused questions that targeted each of the identified competencies. Three or four questions were developed for each competency.

Once these questions were in final form, the SME panel developed a behaviorally anchored rating scale (BARS) for each competency included in the profile. First, each panel member prepared a series of responses to each question. They provided a response that they believed to exemplify each level of performance on a generic five-point scale. We recorded and posted each response on white board in the room. Responses were placed under the scale point for which they had been prepared. We then asked the SMEs to evaluate the responses and negotiate their proper placement in the scale. This process was repeated for each competency in the profile. Thirty-two exemplars were collected for each scale-point value for each competency. We then conducted an affinity analysis, a common quality management technique, to identify common themes and behavioral elements anchored to each level of performance.

A typical BARS method assigns values to descriptors based on their average rating. In contrast, we took a modal approach--we clustered numerical outliers with the pack, provided that they were consistent in meaning. For example, if four descriptors had essentially the same meaning, but one received a different numerical rating, we clustered it into the group with which it had the greatest affinity. The cluster received the ordinal scale value assigned to the majority of descriptors in the cluster. This approach is generally more appropriate for ordinal data and it produces results that makes more sense to the users.

Our next step was to assign descriptive labels to each scale point of each competency-based scale. These descriptive labels were based on the behavior that had been exhibited at that level. These descriptive labels were then shown to the SME focus group for final consensus.

As a final step, we validated the rating scales against a sample of incumbents for whom we had prior performance ratings. We interviewed 11 GBI Agents using the questions that the SMEs had selected and scored their responses using our competency-based scales. As in the CSEA project, these scales identified 80% of the high performers. One low performer and one high performer each were misidentified. Based on the validation data, we further revised the scales to remove overlap, duplication, and ambiguities among the levels.

Further Applications of CBBI

Two new CBBI projects are underway at the Merit System. Both incorporate slight modifications to the process we described earlier. We are preparing a CBBI for the Department of Audits using competencies and interview questions identified and developed by a SME panel. In this project, however, we have the opportunity to base our BARS on data collected from behaviorally focused interviews with incumbents who have been classified by their performance levels. We plan to cross-validate the resulting scales on a second sample.

A similar project to CSEA is underway for the Child Support Enforcement Managers (CSEM). For this project, a SME focus group will develop a tentative competency model. We will then administer a BEI to high performers only. Based on the responses to that interview, we will refine our model, construct our scales, and develop the CBBI questions. The CBBI will then be validated in a sample of incumbents who have been classified according to their level of performance. Incumbents in this sample will represent the full range of performance levels.

A Flexible Methodology

Depending on the resources of the agency and the test developers, this methodology can be modified in to streamline the development process.

Spencer, L. M., & Spencer, S. (1993). Competence at work: Models for superior performance. New York: Wiley.

Charles, Jeff, and Jason develop selection processes for the Georgia Merit System. They can be reached by email at the following addresses. Charles Brooks, Sr. Human Resources Consultant: bro@gms.state.ga.us; Jeff Maile, Advanced Personnel Analyst: jmm@gms.state.ga.us; Jason Fehr, Senior Personnel Analyst: jfehr@gms.state.ga.us.