Conducting a hypothesis test is by nature an uncertainbusiness. Just be cause you do not reject the null hypothesis, itdoes not mean that the null hypothesis is true. There is in factinsufficient evidence from the hypothesis test to mean that you canreject the null hypothesis. If for example, you conducted a 10% test,then even if the null hypothesis we true, 10% of the time using thistest, you would conclude that the null hypothesis we false.
Conversely, if the null hypothesis is false, there is anon zero probability of accepting the null hypothesis when the nullhypothesis is false.
A Type I Erroris made if the null hypothesis
isrejected when it is true.
The probability of making a Type I Error is equal tothe significance level.
A Type II Erroris made if the null hypothesisisaccepted when it is false.
The power of a statistical test is the probability thatthe test will reject the null hypothesis when the null hypothesis isfalse (i.e. that it will not make a Type II error). More powerfultests are more useful, but working out the power of a test can becomplicated.
We must minimise the probability of making a type Ierror and we can do this by reducing the significance level of thetest since this is the probability of making a Type I Error, but thenthe probability of making a Type II Error is increased. In fact wecannot simultaneously reduce both the probability of a Type I Errorand a Type II Error, and a balance between these two evils is alwaysstruck. A lot depends on which is the greater evil – on theconsequences of falsely rejecting a null hypothesis, or falselyaccepting a null hypothesis.
It may be useful to think of the null hypothesis assomething like: The accused is always innocent. Then the alternativehypothesis as being: The accused is guilty. The prosecution mustalways prove the null hypothesis - guilt of the accused, but in factthe alternative hypothesis is never proved in a statisticalhypothesis test. The probability that comes out of a hypothesis test,if the probability method is used, is the probability that the nullhypothesis is true, so that the null hypothesis is only rejected ifthis probability is below the significance level of the test.