Why the Null Hypothesis is Not Accepted (4 of 5)
Assume the experiment measured "well being" on a 50 point scale (with higher scores representing more well being) that has a standard deviation of 10. Further assume the 99% confidence interval computed from the experimental data was:
-0.5 ≤ µ1- µ2 ≤ 1
This says that one can be confident that the mean "true" drug treatment effect is somewhere between -0.5 and 1. If it were -0.5 then the drug would, on average, be slightly detrimental; if it were 1 then the drug would, on average, be slightly beneficial. But, how much benefit is an average improvement of 1? Naturally that is a question that involves characteristics of the measurement scale. But, since 1 is only 0.10 standard deviations, it can be presumed to be a small effect. The overlap between two distributions whose means differ by 0.10 standard deviations is shown below. Although the blue distribution is
slightly to the right of the red distribution, the overlap is almost complete.