Machine learning algorithms rely on their ability to evaluate the constructed hypotheses for choosing the optimal hypothesis during learning and assessing the quality of the model afterwards. Since these estimates, in particular the former ones, are based on the training data from which the hypotheses themselves were constructed, they are inevitably optimistic. The paper shows three different solutions; two for the artificial boundary cases with the smallest and the largest optimism and a general correction procedure called extreme value correction (EVC) based on extreme value distribution. We demonstrate the application of the technique to rule learning, specifically on estimating classification accuracy of rules, and evaluate it on an artificial data set and on a number of UCI data sets. We observed that the correction successfully improves the accuracy estimates. In the last part of evaluation, we describe an approach for combining rules into a linear global classifier and show that using EVC estimates leads to more accurate classifiers.

Paper submitted to journal.

• Source code of Orange 3 add-on