Statistics: Tips and Enigmas
Chuck Schultz
Statistics Tip:
Do you include "omits" in an item analysis?
Item analysis compares performance on individual test questions with a criterion. An independent measure of the dimension in question makes the ideal criterion. Frequently, you have no independent measure handy. In that case, total score for other questions on the test provides an alternative. You can defend using total score when you argue that the other questions are valid measures of the same dimension.
What if you intend to measure several dimensions? Then using total score as a criterion will obscure the value of the item for the specific dimension. You may wind up with items that test the dimension superficially and throw out sensitive measure of the dimension. You may defend using score on the different dimensions as criteria for the items within those dimensions.
Whether you should include omits depends upon what theat tells you about the test question. Did the question stump candidates and so they skipped it? Then comparing how these candidates did on the criterion is relevant information. On the other hand, perhaps the other questions gave the candidates so much trouble they didn't get to this one. In that case, information that candidates omitted the item tells nothing about candidates' dilemma with the item.
Therefore, include "omit" tallies for skipped items but not for those which the candidate did not attempt. You can tell the difference by seeing whether a candidate attempted the later items. Can your computer program tell the difference?
If a candidate omits a large number of items, remove the case from all or part of the analysis. Candidates will likely get low scores if they omit a lot of items. Discrimination indexes will be artificially high for items late in the test if low-scoring candidates omit those items. The indexes will erroneously show these ignored items as better than others that were equitably appraised. What if the final selection measured a different dimension than the selections contribution most to total score?
Statistics Enigma:
The more you gamble the more likely you are to lose...
When you play poker with your friends, your likelihood of winning depends upon your relative skill. In that case, your skill may well improve the more you play. But when you play at an honest casino (we're being hypothetical here) the probability of the payoff is predetermine, and it favors the house. You still have a chance to come out ahead, but it gets more slender the more you play.
Each aspect of each games has its probability of payoff. Let's say you put your money on black at the roulette table. Your chance of winning is 18 of 38 or 0.4737. Eighteen numbers are black, eighteen numbers are red, and two are green (0 and 00). But black pays off two to one, so your chance of coming out ahead is 0.94737 to 1.00. If you play an infinite number of times, you will win back nearly 95 percent of what you spend. If you play an infinite number of times, the chance you will come out ahead is as close to zero as you can get.
The percentage of your money black will pay back for a given number of tries is distributed as the normal curve. The mean of the distribution is 94.737 percent. If you are in the part of the distribution that falls above 100 percent, you will come home a winner. The standard error of the percentage depends upon the probability of winning on each play and the number of times you play: 
, where p is the probability you win (.4737), q the probability you lose (.5236), and n the number of times you play. So as n gets larger, 
gets smaller.
When n is small, a good size tail of the curve extends above 100 percent, so there is a good chance you'll win. When n = 90, one standard error falls 5.263 percentage points above the mean; right at 100%. Therefore, if you play 90 times your chances are nearly one in six that you will go home with more money than you came with (0.1587 of the cases fall above one 
). But then, five times in six you'll be a loser.
In you play 487 times, 100 percent of the money you started with falls 2.326 standard error above the mean, and your chances of going home richer are one in one hundred. Playing 860 times makes your chances of coming out ahead one in a thousand. If the payoff doubles your one dollar on 47.37 percent of the 860 times (407 times), you will have lost $46.00.
If you put $860.00 on black all at once, there is a 47 percent chance you'll walk off with $1720.00 -- and a 53 percent chance you will be out $860.00. With each succeeding play you increase the likelihood that you will finish in the hole. The best way to come out ahead is to figure out how much you are willing to lose and play it all at once.
But your luck has to change. If you have lost ten straight times, you're due to win. Isn't that the law of averages? No, the formal name of that statement is The Gambler's Fallacy. It doesn't matter how many times in a row red has come up, the probability that black will come up next is .4737. The law of averages say that, once you are a thousand dollars in the hole, you will probably keep descending, as long as you play, at the same rate as everybody else.
People frequently come back from a weekend at the casino and announce their winning experiences. How can so many people revel in such good luck? It's simple. They keep poor records. When you get greatly in the hole, throw away your records and start over. Just remember the big payoffs and don't count how much you lost beforehand.
Not all games are like roulette where you can count on your chances. Some games have a strategy - so you can make the chances of payoff worse. In video poker, you can come closest to keeping what you came with by playing an optimum strategy. In craps or twenty-one, the wise player improves the odds to nearly even. When a player becomes too wise, the casino closes the games to her or him. The moral of the tale is: keep your winnings small.
You would rather take advice from someone with a lot of gambling experience, right? Like if you want advice on marriage, you get advice from someone who has been married several times? Well, if someone has been to the casino very many times, I would certainly be wary of advice from that quarter.
Chuck Schultz may be reached at (360) 923-5340, 2941 B Firwood Loop, Olympia, WA 98501-4844.
© Copyright 1997 by the IPMA Assessment Council. All rights reserved.