Relationships Between Support Vector Classifiers and Generalized Linear Discriminant Analysis on Support Vectors

Abstract

The linear discriminant analysis based on the generalized singular value decomposition (LDA/GSVD) has recently been introduced to circumvents the nonsingularity restriction that occur in the classical LDA so that a dimension reducing transformation can be effectively obtained for undersampled problems. In this paper, relationships between support vector machines (SVMs) and the generalized linear discriminant analysis applied to the support vectors are studied. Based on the GSVD, the weight vector of the hard margin SVM is proved to be equivalent to the dimension reducing transformation vector generated by LDA/GSVD applied to the support vectors of the binary class.It has also been shown that dimension reducing transformation vector and the weight vector of soft margin SVMs are related when a subset of support vectors are considered. These results can be generalized when kernelized SVMs and the kernelized KDA/GSVD are considered.Illustrating the relationship, it is shown that a classification problem can be interpreted as a data reduction problem.