Significance Testing and Confidence Intervals

Prerequisites

• Confidence Intervals, Introduction to Hypothesis Testing, Significance Testing

Questions

• How to determine from a confidence interval whether a test is significant?
• Why a confidence interval makes clear that one should not accept the null hypothesis?

• There is a close relationship between confidence intervals and significance tests
• Specifically, if a statistic is significantly different from 0 at the 0.05 level then the 95% confidence interval will not contain 0
• All values in the confidence interval are plausible values for the parameter whereas values outside the interval are rejected as plausible values for the parameter
• In the Physicians' Reactions case study, the 95% confidence interval for the difference between means extends from 2.00 to 11.26. Therefore, any value lower than 2.00 or higher than 11.26 is rejected as a plausible value for the population difference between means
• Since zero is lower than 2.00, it is rejected as a plausible value and a test of the null hypothesis that there is no difference between means is significant
• It turns out that the p value is 0.0057. There is a similar relationship between the 99% confidence interval and Significance at the 0.01 level

• Whenever an effect is significant, all values in the confidence interval will be on the same side of zero (either all positive or all negative). Therefore, a significant finding allows the researcher to specify the direction of the effect
• There are many situations in which it is very unlikely two conditions will have exactly the same population means

• For example, it is practically impossible that aspirin and acetaminophen provide exactly the same degree of pain relief.
• Therefore, even before an experiment comparing their effectiveness is conducted, the researcher knows that the null hypothesis of exactly no difference is false
• However, the researcher does not know which drug offers more relief
• If a test of the difference is significant, then the direction of the difference is established because the values in the confidence interval are either all positive or all negative.
• If the 95% confidence interval contains zero (more precisely, the parameter value specified in the null hypothesis), then the effect will not be significant at the 0.05 level
• Looking at non-significant effects in terms of confidence intervals makes clear why the null hypothesis should not be accepted when it is not rejected: Every value in the confidence interval is a plausible value of the parameter
• Since zero is in the interval, it cannot be rejected
• However, there is an infinite number of values in the interval (assuming continuous measurement), and none of them can be rejected either.

Questions

1 The null hypothesis for a particular experiment is that the mean test score is 20. If the 99% confidence interval is (18, 24), can you reject the null hypothesis at the 0.01 level?

	Yes
	No

Answer >>

You cannot reject the null hypothesis because the confidence interval shows that 20 is a plausible population parameter.

2 Select all that apply. Which of these 95% confidence intervals for the difference between means represents a significant difference at the 0.05 level?

	(-4.6, -1.8)
	(-0.2, 8.1)
	(-5.1, 6.7)
	(3.0, 10.9)

Answer >>

This study is testing the difference between means, and significant differences would be either larger or smaller than 0. Thus, confidence intervals that do not contain 0 represent statistically significant findings.

3 If a 95% confidence interval contains 0, so will the 99% confidence interval.

	True
	False

Answer >>

The 99% confidence interval contains all of the values that the 95% confidence interval has, but it extends farther at both ends and has other values, too. If something is not significant at the 0.05 level, it is also non-significant at the 0.01 level.

4 Select all that apply. A person is testing whether a coin that a magician uses is biased. After analyzing the results from his coin flipping, the p value ends up being 0.21, so he concludes that there is no evidence that the coin is biased. Based on this information, which of these is/are possible 95% confidence intervals on the population proportion of times heads comes up?

	(0.43, 0.55)
	(0.32, 0.46)
	(0.48, 0.64)
	(0.76, 0.98)
	(0.81, 1.33)

Answer >>

Because the p value was 0.21, we know that the 95% confidence interval contains the null hypothesis parameter, 0.5. Thus, both of the confidence intervals that contain 0.5 are possible confidence intervals that this researcher could have computed.