Example Calculations
Prerequisites
- Introduction to Power, Binomial Distribution, Testing a Single Mean
Compute power using the binomial distribution
In the "Shaking and Stirring Martinis" case study, the question was whether Mr. Bond could tell the difference between martinis that were stirred and martinis that were shaken. For the sake of this example, assume he can tell the difference and is able to correctly state whether a martini had been shaken or stirred 0.75 of the time. Now, suppose an experiment is being conducted to investigate whether Mr. Bond can tell the difference. Specifically is Mr. Bond correct more than 0.50 of the time. We know that he can (that's an assumption of the example). However, the experimenter does not know and asks Mr. Bond to judge 16 Martinis. The experimenter will do a significance test based on the binomial distribution. Specifically, if a one tailed test is significant at the 0.05 level then he or she will conclude that Mr. Bond can tell the difference. The probability value is computed assuming the null hypothesis is true (π = 0.50). Therefore, the experimenter will determine how many times Mr. Bond is correct, and compute the probability of being correct that many or more times given that the null hypothesis is true. The question is: what is the probability the experimenter will correctly reject the null hypothesis that π = 0.50? In other words, what is the power of this experiment.
The binomial distribution for N = 16 and π = 0.50 is shown in Figure 1. The probability of being correct on 11 or more trials is 0.105 and the probability of being correct on 12 or more trials is 0.038. Therefore, the probability of being correct on 12 or more trials is less than 0.05. This means that the null hypothesis will be rejected if Mr. Bond is correct on 12 or more trials and will not be rejected otherwise.
Figure 1. The binomial distribution for N = 16 and π = 0.50.
We know that Mr. Bond is correct 0.75 of the time. (Obviously the experimenter does not know this or there would be no need for an experiment.) The binomial distribution with N = 16 and π = 0.75 is shown in Figure 2.
Figure 2. The binomial distribution for N = 16 and π = 0.75.
The probability of being correct on 12 or more trials is 0.63. Therefore, the power of the experiment is 0.63.
To sum up, the probability of being correct on 12 or more trials given that the null hypothesis is true is less than 0.05. Therefore, if Mr. Bond is correct on 12 or more trials, the null hypothesis will be rejected. Given Mr. Bond's true ability to be correct on 0.75 of the trials, the probability he will be correct on 12 or more trials is 0.63. Therefore power is 0.63.
Compute power using the normal distribution
In the section on Testing a Single Mean for significance, the first example was based on the assumption that the experimenter knew the population variance. Although this is rarely true in practice, the example is very useful for pedagogical purposes. For the same reason, the following example assumes the experimenter knows the population variance. Power calculators are available for situations in which the experimenter does not know the population variance.
Suppose a math achievement test were known to have a mean of 75 and standard deviation of 10. A researcher is interested in whether a new method of teaching results in a higher mean. Assume that although the experimenter does not know it, the population mean for the new method is 80. The researcher plans to sample 25 subjects and do a one-tailed test of the whether the sample mean is significantly higher than 75. What is the probability that the researcher will correctly reject the false null hypothesis that the population mean is 75? The following shows how this probability is computed.
The researcher assumes that the population standard deviation with the new method is the same as with the old method (10) and that the distribution is normal. Since the population standard deviation is assumed to be known, the researcher can use the normal distribution rather than the t distribution to compute the p value. Recall that the standard error of the mean (σM) is
which is equal to 10/5 = 2 in this example. As can be seen in Figure 3, if the null hypothesis that the population mean equals 75 is true, then the probability of a sample mean being greater than or equal to 78.29 is 0.05. Therefore, the experimenter will reject the null hypothesis if the sample mean, M, is 78.29 or larger.
Figure 3. The sampling distribution of the mean if the null hypothesis is true. (figure created with the Inverse Normal Calculator)
The question, then, is what is the probability the experimenter gets a sample mean greater than 78.29 given that the population mean is 80? Figure 4 shows that this probability is 0.80.
Figure 4. The sampling distribution of the mean if the population mean is 75. The test is significant if the sample mean is 78.29 or higher. (figure created with the Normal Calculator)
Therefore, the probability that the experimenter will reject the null hypothesis that the null hypothesis is 75 is 0.80. In other words, power = 0.80.
Use a power calculator to compute power for the t distribution
Calculation of power is more complex for t-tests and for Analysis of Variance. The power calculator computes power for a t test of independent groups. Calculators for other types of designs can be found at this
Russ Lenth's Power Calculators (external link)
Questions
