Misconceptions

Prerequisites

• Introduction to Hypothesis Testing, Statistical Significance, Type I and II Errors

Questions

• Why the probability value is not the probability the null hypothesis is false?
• Why a low probability value does not necessarily mean there is a large effect?
• Why a non-significant outcome does not mean the null hypothesis is probably true?

Misconception 1: The probability value is the probability that the null hypothesis is false

• The probability value is the probability of a result as extreme or more extreme given that the null hypothesis is true
• It is the probability of the data given the null hypothesis
• It is NOT the probability that the null hypothesis is false.

Misconception 2: A low probability value indicates a large effect

• A low probability value indicates that the sample outcome (or one more extreme) would be very unlikely if the null hypothesis were true. A low probability value can occur with small effect sizes, particularly if the sample size is large.

Misconception 3: A non-significant outcome means that the null hypothesis is probably true

• A non-significant outcome means that the data do not conclusively demonstrate that the null hypothesis is false

Questions

1 A low probability value indicates a large effect.

	True
	False

Answer >>

A low probability value indicates that the sample outcome (or one more extreme) would be very unlikely if the null hypothesis were true. A low probability value can even occur with small effect sizes, particularly if the sample size is large.

2 The probability value represents the probability of the null hypothesis given the data.

	True
	False

Answer >>

The probability value represents the probability of the data given the null hypothesis.

3 A non-significant outcome means that the data do not conclusively demonstrate that the null hypothesis is false.

	True
	False

Answer >>

This is true, but a common misconception is that a non-significant outcome means that the null hypothesis is probably true.

4 The probability value is the probability that the null hypothesis is false.

	True
	False

Answer >>

The probability value is the probability of a result as extreme or more extreme given that the null hypothesis is true. It is the probability of the data given the null hypothesis. It is not the probability that the null hypothesis is false.