Statistics for Decision Makers - 05.05 - Probability - Base Rate Fallacy

title

05.05 - Probability - Base Rate Falacy

author

Bernard Szlachta (NobleProg Ltd) bs@nobleprog.co.uk

Drunk Driving。

• Breathalysers display a false result in 5% of the cases tested
• They never fail to detect a truly drunk person
• 1/1000 of drivers are driving drunk
• Policemen then stop a driver at random, and test them
• The breathalyser indicates that the driver is drunk

How high is the probability the driver is really drunk?

Result\Reality	Drunk	Sober
Test Positive	1	0.05
Test Negative	0	0.95
Historic Data	0.001	0.999

Drunk Driving。

Let us assume we tested 1,000,000 people.

True Condition
Drunk		Not Drunk
1,000		999,000
Positive	Negative	Positive	Negative
1,000	0	49,950	949,050

• How many of those people tested positive?
• How many of those who tested positive were really drunk?
• What is a "favourable" outcome?
• How many "favourable" outcomes are there?
• How many possible outcomes are there?
• What is the probability of a person who is tested being really drunk?

Drunk Driving。

Let us assume we tested 1,000,000 people.

True Condition
Drunk		Not Drunk
1,000		999,000
Positive	Negative	Positive	Negative
1,000	0	49,950	949,050

How many of those people tested positive?

1,000 + 49,950 = 50,950

How many of those who tested positive were really drunk?

1000

What is the probability of a person who is tested being really drunk?

1000/50,950=0.01962

Drunk Driving。

Base Rate Fallacy。

• The Base Rate in our case is 0.001 and 0.999 probabilities.
• An overwhelming proportion of people are sober, therefore the probability of a false positive (5%) is much more prominent than the 100% probability of a true positive.
• People tend to simply ignore the base rates, hence it is called (base rate neglect).
• In other words, no matter what the base rates, people tend to look at only the "test accuracy rate".

Base Rate Fallacy Examples。

• Detecting terrorists
• Detecting a rare disease
• Detecting prospective customers (provided that most people will not buy our product)
• Some DNA tests

Bayes theorem。

What is the probability that a driver is drunk given that the breathalyser indicates that he/she is drunk?

$P(Drunk|Positive)$

Bayes' Theorem tells us that:

$P(Drunk|Positive)={\frac {P(Positive|Drunk)\,P(Drunk)}{P(Positive|Drunk)\,P(Drunk)+P(Positive|Sober)\,P(Sober)}}$

We were told the following in the first paragraph:

$P(Drunk)=0.001$

$P(Sober)=0.999$

$P(Positive|Drunk)=1.00$

$P(Positive|Sober)=0.05$

After using Bayes' Theorem:

$P(Drunk|Positive)=0.019627\cdot$

Is a promotion really working?。

An online advertising company knows (based on its historical record) that 10% of the people who try the trial version of their services will convert into paying customers.

• You propose to introduce a promotion: each new customer will be granted a free $100 for advertising
• You know that some people will just register to use the $100 even if they do not intend to convert into paying customers
• You want to test the effectiveness of the free $100 promotion
• After running the promotion, 40% of customers who converted used the $100 promotion
• Also, 10% of prospects who did not convert used the promotion

Is the promotion really working?。

Does the promotion increase the probability of conversion?

Events

Prom: Customer uses the promotion
NotProm: Customer does not use the promotion
Con: Customer converts
NotCon: Customer does not convert 

P(Con) = 0.1

An online advertising company knows, based on its historical record, that 10% of the people who try the trial version of their services will convert into customers

P(Prom|Con) = 0.4

After running the promotion, 40% of customers who converted used the $100 promotion

P(Prom|NotCon) = 0.1

Also, 10% of prospects who used the promotion did not convert

Compute complementary probabilities

P(NotCon) = 0.9
P(NotProm|Con) = 0.6
P(NotProm|NotCon) = 0.9

Does the promotion increase probability of conversion?

P(Con|Prom) > P(Con) = 0.1
P(Con|Prom) from BaysTheorm = 0.308

Quiz。

Please find the Quiz here

Quiz

1 The prospective customer conversion rate is 1%

• You want to prioritize prospects who are more likely to convert.
• You create a test, which has an 80% probability of correctly detecting that a prospect will convert into a customer.
• But for 9.6% of prospects, the test misdetects customers, i.e. they will not convert.

A prospect is positively tested, what is the probability they will convert into a customer?

	below 33%
	between 33% and 66%
	above 66%

Answer >>

below 33%

7.8%

2 The prospective customer conversion rate is 50%

• You want to prioritize prospects who are more likely to convert.
• You create a test, which has an 80% probability of correctly detecting that the prospect will convert into a customer.
• But for 9.6% of prospects, the test misdetects customers, i.e. they will not convert.

A prospect is positively tested, what is the probability they will convert into a customer?

	below 33%
	between 33% and 66%
	above 66%

Answer >>

between 33% and 66%

3 The prospective customer conversion rate is 99%.

• You want to prioritize prospects who are more likely to convert.
• You create a test, which has an 80% probability of correctly detecting that the prospect will convert into a customer.
• But for 9.6% of prospects, the test misdetects customers, i.e. they will not convert.

A prospect is positively tested, what is the probability they will convert into a customer?

	below 33%
	between 33% and 66%
	above 66%

Answer >>

above 66%

4 The prospective customer conversion rate is 99%

• You want prioritize prospects who are more likely to convert
• You create a test, which has an 80% probability of correctly detecting that the prospect will convert into a customer
• But for 9.6% of prospects, the test misdetects customers, i.e. they will not convert

According to the text, which of the below is a base rate (select two)?

	1%
	99%
	9.6%
	80%
	20%

Answer >>

1%, 99%