Choosing an a-level

Your choice for a determines the probability of Type I error. The smaller the value, the less likely you are to incorrectly reject the null hypothesis (H0). However, a smaller value for a also means lower power, and a decreased chance of detecting an effect if one exists.

By convention, the most commonly used a-level is 0.05. With an a = 0.05, your chance of finding an effect that does not really exist is only 5%. In most situations, this probability of error is considered acceptable. However, when selecting a for a particular test, you may want to ask yourself which mistake would be worse, finding an effect that does not really exist, or not finding an effect that really does exist.

Choose a smaller a. Sometimes it may be better to choose a smaller, more conservative value for a. For example, suppose you are testing samples from a new milling machine, and trying to decide whether to purchase and install a dozen of the machines in your processing plant. If the new machine is more accurate than the ones you are currently using, you stand to save a fair amount of money because you will be producing fewer defective products. However, the cost of purchasing and installing a dozen machines is very high. You want to be sure that the new machine is more accurate before making the purchase. In this case, you might want to select a lower value for a, such as 0.001. That way, there is only a 0.1% chance that you will conclude the new machine is more accurate, if in fact it is not.

Choose a larger a. On the other hand, sometimes it is better to choose a larger, more liberal value for a. For example, suppose you are a jet engine manufacturer and you are testing the stability of a new, less expensive ball bearing. Obviously, saving a small amount of money on ball bearings does not outweigh the potentially disastrous effects if the bearings are not sound. Therefore, you might want to select a higher value for a, such as 0.1. Although this means you will be more likely to mistakenly conclude there is a difference if none really exists, more importantly, you will also be more likely to detect a difference in the stability of the bearings if one does exist.