On optimality in statistical modeling: Optimal classifier design and a hierarchical modeling of the identification problem

Wei-Tsih Lee, Purdue University

Abstract

The main emphasis of this research has been on the analysis of the complexity issue in statistical modeling. We have achieved this goal by examining its implications for optimal classifier design and for the hierarchical modeling approach to the identification problem. In the pattern recognition problem, we start from an analysis of classification error. In general, the error probability of a classifier can be bounded from above by the sum of empirical and generalization errors. The first is the error observed on training samples, and the second is the discrepancy between the error probability and the empirical error. Since both the terms depend on classifier complexity, we therefore need a proper measure for classifier complexity. In this research, we adopt the Vapnik and Chervonenkis dimension (VCdim) as such a measure. Based on this complexity measure, we develop an estimate for generalization error. An optimal classifier design criterion, the Generalized Minimum Empirical Criterion (GMEE), has been proposed. We prove that the GMEE criterion is $\Gamma$-optimal, which means that the criterion can select the best classifier from $\Gamma$, a collection of classifiers with finite VCdim. The GMEE criterion is then applied to the design of optimal neural network classifiers. Experimental results show good performance on two examples. In the hierarchical modeling approach to the identification problem, we endeavor to construct the simplest model that is consistent with the data. Our approach is to generate a hierarchical model by combining simple models. A search method based on Simulated Annealing is developed to evaluate the possible combinations. A novel feature of this research is that the candidate models are evaluated by the SEC, which is a variant of the Minimum Description Length (MDL) criterion. Experiments involving the identification of chaotic time series show that the method can provide not only the best approximate model but also a set of suboptimal models which can be adopted when only limited resources available. (Abstract shortened with permission of author.)

Degree

Ph.D.

Advisors

Tenorio, Purdue University.

Subject Area

Electrical engineering

COinS