Some Practical Questions about Assessment Centers

by Deonda Scott

As assessment professionals, we often have to answer candidates' practical questions about various tests and make decisions about how to maintain or improve assessment quality with limited time and resources. Recently, I decided to collect and analyze some data to answer questions about our own processes. Fair Warning: there are some measurement problems inherent in this data. For example: data from four different assessment center administrations across seven years is pooled for several of the comparisons, and subject numbers are low. Despite these problems, I hope these results are helpful- if for no other purpose than getting you interested in running some numbers and sharing your own data. I'd love to hear about it!

Setting: City Fire Department. Candidates are either Engineers or Firefighters who have passed the Engineer's promotional exam and are attempting to qualify for the Lieutenant Promotional eligibility list. The promotional exam consists of a written test used as a screener, an emergency incident command simulation, and an assessment center. The promotional exam is conducted once every two years. These data are a combination of four assessment centers conducted between 1989 and 1995.

Question: "I've heard that people generally don't do well on their first assessment center."

Not true: The average score of a person taking the assessment center for the first time is 71.41 (SD = 8.74, n=51). The average score of a person taking the assessment center for the second, third, or fourth time is 72.63, (SD = 9.46, n=27).

Question: "I didn't do well on my first assessment center, but now I know what the assessors are looking for. I'll do better the second time, right?"

Maybe, maybe not. The average first score of individuals who went through the assessment center at least twice is 71.53 (SD = 5.73, n=15), and the average second score is 70.58, (SD = 7.99, n=15). Of those who took the assessment center a second time, ten individuals' scores decreased and five increased. Of those whose score increased, the average increase in score was 11.86 (SD=6.98, n=5), and of those whose score went down, the average decrease was 7.35 (SD=3.87, n=10). The correlation between an individual's first score and second score was r = -.16 (n=15).

There are many factors that may explain this correlation. Those who performed well on the assessment center the first time were promoted during the life of the list and did not take the assessment center again. In addition, individuals who failed or did very poorly tended not to go through the process a second time. Those who take the process more than once have at least two years between assessments. Therefore, this correlation should not be viewed as a test-retest reliability estimate. One possible explanation for the correlation is based on comments by candidates. Although some may try to develop their skills as suggested, many view the process as a game whose rules they need to learn. They often listen to advice from others which sometimes backfires because they are trying to "act" rather than perform effectively.

Question: "The same assessment center exercises were used twice. Didn't those who went through the same assessment center last time have an advantage over those who've never been through an assessment center?"

Intuitively it would seem that anyone who had been given developmental feedback on their assessment center performance, then given the same exercises two years later, would have an edge over those who had never been through an assessment center before. However, the average score of those who were going through the assessment center for the first time was 69.72 (SD=6.72, n=14) and the average score of those who had gone through the same assessment center exercise two years prior was 73.02 (SD=8.98, n=9), a difference of 3.26 points - less than one half of a standard deviation.

Question: "I took these same assessment center exercises in 1993, but my score went down in 1995. How did other repeat takers do?"

Of the nine people who took the same assessment center exercises in 1993 and 1995, four improved their score, and five got lower scores in 1995. The average first score of individuals and was 71.77 (SD=4.92, n=9), and the average second score of those same individuals was 73.02 (SD=9.20, n=9).

Whew! I actually got through a whole series of comparisons without once testing for significance. It was difficult, but intentional. Candidates are rarely impressed with "statistically significant" differences, they prefer practical differences. There are no practical differences presented here, although some of these results may run counter to what we might expect.

On the whole, it seems that those taking the assessment center for the first time have every right to be optimistic that they will be able to do well. One possible inference from these results is that we may not have to rewrite assessment center exercises after each use even if some candidates will be seeing them for the second time. But keep in mind that this finding may have more to do with candidates' perceptions of the author. The following comment was made by a candidate who had been through the same exercises twice: "I worked through everything just like the first time. She's so tricky, I figured she might have slipped in some little things that changed the whole exercise."

Deonda Scott is a Test Validation Specialist with the City of Orlando. She can be reached at: City of Orlando, 400 S. Orange Ave., Orlando FL 32801. (407) 2462061 or by email at dscott@ci.orlando.fl.us.