Non-Significant Results

Prerequisites

• Introduction to Hypothesis Testing, Significance Testing, Type I and II Errors

Questions

• What does it mean to accept the null hypothesis?
• Why the null hypothesis should not be accepted?
• How a non-significant result can increase confidence that the null hypothesis is false?
• What are the problems of affirming a negative conclusion?

• When P-value is high, it means that the data provide little or no evidence that the null hypothesis is false
• The high p-value is not evidence that the null hypothesis is true
• The problem is that it is impossible to distinguish a null effect from a very small effect

• For example, in the James Bond Case Study, suppose Mr. Bond is, in fact, just barely better than chance at judging whether a Martini was shaken or stirred
• Assume he has a 0.51 probability of being correct on a given trial (π = 0.51)
• Let's say Experimenter Jones (who did not know π = 0.51) tested Mr. Bond and found he was correct 49 times out of 100 tries. * How would the significance test come out? The experimenter’s significance test would be based on the assumption that Mr. Bond has a 0.50 probability of being correct on each trial (π = 0.50). Given this assumption, the probability of his being correct 49 or more times out of 100 is 0.62
• 0.62 is far higher than 0.05
• This result, therefore, does not give even a hint that the null hypothesis is false
• However, we know (but Experimenter Jones does not) that π = 0.51 and not 0.50 and therefore that the null hypothesis is false
• So, if Experimenter Jones had concluded that the null hypothesis were true based on the statistical analysis, he or she would have been mistaken

Concluding that the null hypothesis is true is called accepting the null hypothesis. To do so is a serious error.

Questions

1 You have just analyzed the results from your experiment, and you calculated p = 0.13. What conclusions can you make? Select all that apply.

	You reject the null hypothesis.
	You accept the null hypothesis.
	You fail to reject the null hypothesis.
	You accept the alternative hypothesis.

Answer >>

You are unable to reject the null hypothesis or accept the alternative hypothesis if your p value is 0.13. However, you cannot conclude that the null hypothesis is true either. Thus, you only fail to reject the null hypothesis.

2 You have just given a group of 2nd graders and 1st graders a reading test. You found that the 2nd graders performed better than the 1st graders, but you calculated a p value of .08, which was not significant at the .05 level. After getting these results, what should your thoughts be about the difference between 1st and 2nd graders on this reading test?

	You are more confident that there is a difference.
	You are less confident that there is a difference.
	You now know that the difference is actually zero.

Answer >>

Although you were unable to reject the null hypothesis here, you did find a difference in your sample. Because of this sample difference, you can now be more confident that the population difference does really exist, and doing further research is the best way to find out. You definitely do not accept the null hypothesis.