Significant Results

Prerequisites

• Introduction to Hypothesis Testing, Statistical Significance, Type I and II Errors, One and Two-Tailed Tests

Questions

• Should rejection of the null hypothesis should be an all-or-none proposition?
• What is the value of a significance test when it is extremely likely that the null hypothesis of no difference is false even before doing the experiment?

Interpreting Significant Results

• When a probability value is below the α level, the effect is statistically significant and the null hypothesis is rejected
• However, not all statistically significant effects should be treated the same way
• For example, you should have less confidence that the null hypothesis is false if p = 0.049 than p = 0.003
• Thus, rejecting the null hypothesis is not an all-or-none proposition

If the null hypothesis is rejected, then the alternative hypothesis is accepted

Interpreting results of one-tailed test

Consider the one-tailed test in the James Bond case study:

• Mr. Bond was given 16 trials on which he judged whether a Martini had been shaken or stirred and the question is whether he is better than chance on this task

H 0 π ≤ 0.5
π is the probability of being correct on any given trial

• If this null hypothesis is rejected, then the alternative hypothesis that π > 0.5 is accepted
• If π is greater than 0.50 then Mr. Bond is better than chance on this task

Interpreting results of two-tailed test

Now consider the two-tailed test used in the Physicians' Reactions case study

H0: μobese = μaverage
H1: μobese < μaverage
or
H1: μobese > μaverage

• The direction of the sample means determines which alternative is adopted
• If the sample mean for the obese patients is significantly lower than the sample mean for the average-weight patients, then one should conclude that the population mean for the obese patients is lower than than the sample mean for the average-weight patients

• There are many situations in which it is very unlikely two conditions will have exactly the same population means
• For example, it is practically impossible that aspirin and acetaminophen provide exactly the same degree of pain relief
• Therefore, even before an experiment comparing their effectiveness is conducted, the researcher knows that the null hypothesis of exactly no difference is false
• However, the researcher does not know which drug offers more relief
• If a test of the difference is significant, then the direction of the difference is established

Can we really tell which population mean is larger?

This text is optional

• Some textbooks have incorrectly stated that rejecting the null hypothesis that two population means are equal does not justify a conclusion about which population mean is larger
• Instead, they say that all one can conclude is that the population means differ
• The validity of concluding the direction of the effect is clear if you note that a two-tailed test at the 0.05 level is equivalent to two separate one-tailed tests each at the 0.025 level
• The two null hypotheses are then

μobese ≥ μaverage
μobese ≤ μaverage

• If the former of these is rejected, then the conclusion is that the population mean for obese patients is lower than that for average-weight patients
• If the latter is rejected, then the conclusion is that the population mean for obese patients is higher than that for average-weight patients

Questions

1 Which of the following probability values gives you the most confidence that the null hypothesis is false?

	p = 0.28
	p = 0.05
	p = 0.042
	p = 0.003

Answer >>

The probability value is the proportion of times that you would get a difference in your sample as large or larger than the one you found if the null hypothesis were actually true. Thus, lower probability values make you more confident that the null hypothesis is false. In this case, the lowest probability value is 0.003.

2 You are testing the difference between high school freshmen and seniors on SAT performance. The null hypothesis is that the population mean SAT score of the seniors is equal to the population mean SAT score of the freshmen. You randomly sample 20 students in each grade and have them take the SAT. You find that the sample mean of the seniors is significantly higher than the sample mean of the freshmen. Which alternative hypothesis is accepted?

	The population mean SAT score of the seniors is less than the population mean SAT score of the freshmen.
	The population mean SAT score of the seniors is greater than the population mean SAT score of the freshmen.
	You cannot be sure which alternative hypothesis to accept. You just know that the null hypothesis was rejected.

Answer >>

The direction of the sample means determines which alternative is adopted. In this example, the sample means show that seniors performed better, so this alternative is accepted.

3 If you are already certain that a null hypothesis is false, then:

	Significance testing provides no useful information since all it does is reject a null hypothesis.
	Significance testing is informative because you still need to know whether an effect is significant even if you know the null hypothesis is false.
	When a difference is significant you can draw a confident conclusion about the direction of the effect.

Answer >>

A significant result lets you conclude the direction of the result. After a non-significant result, the direction of the difference is uncertain.