Confidence Intervals a Suppose we have 2 classifiers, M, and M2 which one is better? a Use 10-fold cross-validation to obtain error(M,) and error(M2) a These mean error rates are just estimates of error on the true population of future data cases a What if the difference between the 2 error rates is just attributed to chance? o Use a test of statistical significance e Obtain confidence limits for our error estimatesConfidence Intervals ◼ Suppose we have 2 classifiers, M1 and M2 , which one is better? ◼ Use 10-fold cross-validation to obtain error(M1 ) and error(M2 ) ◼ These mean error rates are just estimates of error on the true population of future data cases ◼ What if the difference between the 2 error rates is just attributed to chance? ◆ Use a test of statistical significance ◆ Obtain confidence limits for our error estimates