Statistics for Decision Makers - 12.02 - Testing Means - Repeated Tests

title

12.02 - Testing Means - Repeated Tests

author

Bernard Szlachta (NobleProg Ltd)

Repeated Test。

• Imagine you conducted customer satisfaction tests before and after (pre and post) a change to your web application
• The results were (10,20,30) before the change and (1,2,3,4,5) after the change
• The test was inconclusive
• You administered the test again
• The results were pre: (1,21,31,29) post: (1,2,3,4,5)
• The test was inconclusive again!

What decision should you make?

Test 1 。

 pre <- c(10,20,30)
 post <- c(1,2,3,4,5)
 t.test(pre , post)

p-value = 0.09644
alternative hypothesis: true difference in means is not equal to 0 
sample estimates:
mean of x mean of y 
       20         3

Test 2 。

 pre <- c(1,21,31,29)
 post <- c(1,2,3,4,5)
 t.test(pre , post)

p-value = 0.08283
alternative hypothesis: true difference in means is not equal to 0 
sample estimates:
mean of x mean of y 
     20.5       3.0

Combined Result。

 pre <- c(1,21,31,29,10,20,30)
 post <- c(1,2,3,4,5,1,2,3,4,5)
 t.test(pre , post)

p-value = 0.006545
alternative hypothesis: true difference in means is not equal to 0 
sample estimates:
mean of x mean of y 
 20.28571   3.00000

Combined Results interpretation。

• In the first two tests, the sample size was too small to draw any conclusions
• Despite this, it would be quite unlikely to have results with a P-value close to 5% twice in a row
• Assuming that other things are equal (ceteris paribus), you can simply treat both tests as one test with a bigger sample size
• In this case, the combined results test was very significant (P-value 0.6%)

Quiz

1 During the last year a survey has been conducted every month to check whether there is a significant improvement in job satisfaction among employees. Each time you test whether there is a significant difference between the current and the previous month's results. All tests are inconclusive with a P-value close to 0.05%. What can be assumed?

	There is a difference in satisfaction. It is unlikely that a P-value that small appeared so many times in a row
	You can compare the result from last January to January this year to conclude whether there is a difference in a year
	Nothing can be concluded

Answer >>

You cannot assume that test is inconclusive as each test assumes a different population (the population is the true satisfaction in a given month)