Statistics: Tips and Enigmas
by Chuck Schultz
Statistics Tip:
How Big is a Standard Deviation?
Linda looked up and smiled as I walked by and said, "I got a standard deviation of 65 on the Claims Analyst test."
"You forgot to take the square root," I responded.
She looked quickly at her figures. She blushed and asked, "How did you know?"
I knew it was a multiple-choice test scored one point for each right answer. I figured the test had not more than about 100 questions and the lowest score would be above 50 and the highest score probably not over 90. I could picture a distribution where the difference between the highest and lowest score would give a range of about 40. There is no way you can get a standard deviation of 65 with a range of 40.
In a normal curve, about one sixth of the cases fall more than one standard deviation above the mean, and two percent fall more than 2.05 standard deviations above the mean. If you had 50 cases, you can expect about one of them to be two standard deviations above the mean and one more about two standard deviations below the mean.
So from the lowest score to the highest score will be about four standard deviations. In other words, the standard deviation will be about one-fourth of the range.
The computer doesn't forget to take square roots, but it does believe me if I key in a three digit number that was supposed to be a two digit number. There, I approximate what a quantity should be before I accept a computed value.
Statistics Enigma:
It Was Valid Last Week, Why Not Now?
When you build a register from a valid test, how long does it stay good? That depends partly upon how skillfully we make our selections. The better the selections supervisors make, the quicker the validity diminishes in regard to the candidates on the register. At the same time the insurance agency and the brewery are looking for workers. Other employers hire away premium candidates.
Managers complain that referred candidates with high scores are not as good as their protégé with a much lower score. They may be correct.
The several persons referred from a register constitute a sub-sample whose spread of scores is restricted compared to the entire register. The spread on the register is restricted from the spread of the applicants. Because of restriction in range, a validity coefficient based on the referral sub-sample will be much lower than the validity for all applicants. In addition, if some persons have already been selected from the register, chances increase that the correlation for candidates on the new referral will be negative! Defenders of tests don't usually acknowledge this point when criticizing people for ignoring the test scores.
A test never predicts perfectly; some candidates are over-predicted and some under predicted. That is, some people fall above the regression line and some below. "Rules of Several" allow us to interview candidates, let them demonstrate their competence, or check references to try to determine which of them fall above the line. That is, we hone the selection to try to find those candidates who will perform at least as well as the test predicted.
Everything else being equal, the person scoring higher on a good test will most likely be the better performer. But those making selections have tried their damnedest to make sure everything is not equal. They have been trying to leave the poor-performing high-scoring candidates on the register.
Chuck Schultz may be reached at (306) 923-5340, 2941B Firwood Loop, Olympia, WA 98501
© Copyright 1996 by the IPMA Assessment Council. All rights reserved.