Type I and II Errors
Jump to navigation
Jump to search
Prerequisites
Questions。
- What is Type I and Type II errors?
- How to interpret significant and non-significant differences?
- Why the null hypothesis should not be rejected when the effect is not significant?
Simplification
- Let us assume that null hypothesis is always about something being not different
- E.g.
- new application is as popular as old one (there is not difference in popularity)
- new hardware is as fast as old one (there is no difference in speed)
- drug doesn't cure the disease (makes no difference to patient health)
Overview。
| State of the world | There is no difference | There is a difference |
|---|---|---|
| Null hypothesis (no difference) | Rejected (there is a difference) |
Not Rejected (there is no difference) |
| Error | Type I False Alarm False Positive |
Type II Missed Detection False Negative |
Example 1。
- Patient have a disease
- There is a problem
| State of the world | Patient doesn't have cancer | Patient have cancer |
|---|---|---|
| What we tell the patient | has cancer (H0 rejected) |
no cancer (H0 not rejected) |
| Error | Type I False Alarm False Positive |
Type II Missed Detection False Negative |
Example 2。
- New hardware is different than old one
- Customer is more satisfied with new application than old one
- There is a increase in people buying our product after running Ad-words campaign
| State of the world | No difference between new and old (no difference) | There is a between old and new(difference) |
|---|---|---|
| Null hypothesis (no difference) | Rejected (difference) | Not Rejected (no difference) |
| Error | Type I False detection |
Type II Missed Detection |
Supplier vs Customer
Type I error。
- Type I error is a rejection of a true null hypothesis
- False Positive or False Alarm can be used instead in business world
- Rejecting the null hypothesis is not an all-or-nothing decision
- The Type I error rate is affected by the α level: the lower the α level the lower the Type I error rate
Probability of Type I error。
- It might seem that α is the probability of a Type I error
- However, this is not correct
- Instead, α is the probability of a Type I error given that the null hypothesis is true
You can only make a Type I error if the null hypothesis is true.
Type II error。
- Type II error is failing to reject a false null hypothesis
- Unlike a Type I error, a Type II error is not really an error
- When a statistical test is not significant, it means that the data do not provide strong evidence that the null hypothesis is false
Lack of significance does not support the conclusion that the null hypothesis is true
- Therefore, a researcher would not make the mistake of incorrectly concluding that the null hypothesis is true when a statistical test was not significant
- Instead, the researcher would consider the test inconclusive
- Contrast this with a Type I error in which the researcher erroneously concludes that the null hypothesis is false when, in fact, it is true.
A Type II error can only occur if the null hypothesis is false
- If the null hypothesis is false, then the probability of a Type II error is called β
- The probability of correctly rejecting a false null hypothesis equals 1- β and is called Power
- Power is the probability of being able to find a difference if it really exists
Errors and Decision Making。
- Increasing the significance level, increases the changes of False Alarm and decrease the changes of Miss Detection
- To decrease changes of both errors, the sample size has be increased
What is more serious?。
- False Alarm is usually more serious, as the test is convulsive
- Missed Detection usually is less harmful e.g.
- if patient failed to detected a disease, they can repeat the test
- on the other hand, if the therapy was implemented to a misdiagnosed problem, the consequence can be worse
- When selecting alpha level which related to probability of False Alarm, it is important to keep that in mind what is more harmful:
- If the administering treatment even for a healthy patient is consider cheap and not harmful, but not detecting the disease of sick patient would be dangerous setting significance level (alpha) high (e.g. 5%) is right thing to do
Supplier v Customer。
- We cannot calculate the probability of Type II error without knowing the true state of the world
- Reduction in probability of Type I error will increase the chance of Type II error
Google v Hard Drive provider
- Hard-drives are provided to the cloud by Company X (supplier) to Google
- H0 is the all hard drives have specification as provided by the supplier
- In other words, the hard drives Google buys, was drawn from the population of the hard drives complying with the spec
- Google tests a sample of the hard drives and test whether they are compliant using Hypothesis Testing
- If Type I error occurs, it is to the suppliers detriment, since hard drives are fine, but will be rejected by Google (after testing)
- If Type II error occurs, Google accepts drives which are not up to the standard
- The sample size can be increased, but that will increase the cost of testing as well
- There is a trade-off between the reduction of errors and the cost of sampling
Supplier v Customer Solution。
- Google and Company X (supplier) can set up alpha and beta to the level where the probability of both errors is the same
- This can be achieved by changing the sample size (sometimes decreasing it), and changing significance level (alpha)
Questions
