Significance Testing and Confidence Intervals

From Training Material
Revision as of 17:03, 3 February 2012 by Izabela Szlachta (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Prerequisites

Questions

  • How to determine from a confidence interval whether a test is significant?
  • Why a confidence interval makes clear that one should not accept the null hypothesis?


  • There is a close relationship between confidence intervals and significance tests
  • Specifically, if a statistic is significantly different from 0 at the 0.05 level then the 95% confidence interval will not contain 0
  • All values in the confidence interval are plausible values for the parameter whereas values outside the interval are rejected as plausible values for the parameter
  • In the Physicians' Reactions case study, the 95% confidence interval for the difference between means extends from 2.00 to 11.26. Therefore, any value lower than 2.00 or higher than 11.26 is rejected as a plausible value for the population difference between means
  • Since zero is lower than 2.00, it is rejected as a plausible value and a test of the null hypothesis that there is no difference between means is significant
  • It turns out that the p value is 0.0057. There is a similar relationship between the 99% confidence interval and Significance at the 0.01 level


  • Whenever an effect is significant, all values in the confidence interval will be on the same side of zero (either all positive or all negative). Therefore, a significant finding allows the researcher to specify the direction of the effect
  • There are many situations in which it is very unlikely two conditions will have exactly the same population means


  • For example, it is practically impossible that aspirin and acetaminophen provide exactly the same degree of pain relief.
  • Therefore, even before an experiment comparing their effectiveness is conducted, the researcher knows that the null hypothesis of exactly no difference is false
  • However, the researcher does not know which drug offers more relief
  • If a test of the difference is significant, then the direction of the difference is established because the values in the confidence interval are either all positive or all negative.
  • If the 95% confidence interval contains zero (more precisely, the parameter value specified in the null hypothesis), then the effect will not be significant at the 0.05 level
  • Looking at non-significant effects in terms of confidence intervals makes clear why the null hypothesis should not be accepted when it is not rejected: Every value in the confidence interval is a plausible value of the parameter
  • Since zero is in the interval, it cannot be rejected
  • However, there is an infinite number of values in the interval (assuming continuous measurement), and none of them can be rejected either.

Questions

1 The null hypothesis for a particular experiment is that the mean test score is 20. If the 99% confidence interval is (18, 24), can you reject the null hypothesis at the 0.01 level?

Yes
No

Answer >>

You cannot reject the null hypothesis because the confidence interval shows that 20 is a plausible population parameter.


2 Select all that apply. Which of these 95% confidence intervals for the difference between means represents a significant difference at the 0.05 level?

(-4.6, -1.8)
(-0.2, 8.1)
(-5.1, 6.7)
(3.0, 10.9)

Answer >>

This study is testing the difference between means, and significant differences would be either larger or smaller than 0. Thus, confidence intervals that do not contain 0 represent statistically significant findings.


3 If a 95% confidence interval contains 0, so will the 99% confidence interval.

True
False

Answer >>

The 99% confidence interval contains all of the values that the 95% confidence interval has, but it extends farther at both ends and has other values, too. If something is not significant at the 0.05 level, it is also non-significant at the 0.01 level.


4 Select all that apply. A person is testing whether a coin that a magician uses is biased. After analyzing the results from his coin flipping, the p value ends up being 0.21, so he concludes that there is no evidence that the coin is biased. Based on this information, which of these is/are possible 95% confidence intervals on the population proportion of times heads comes up?

(0.43, 0.55)
(0.32, 0.46)
(0.48, 0.64)
(0.76, 0.98)
(0.81, 1.33)

Answer >>

Because the p value was 0.21, we know that the 95% confidence interval contains the null hypothesis parameter, 0.5. Thus, both of the confidence intervals that contain 0.5 are possible confidence intervals that this researcher could have computed.


Template:Statistics Links