A reproducing kernel Hilbert space formulation of the principle of relevant information

Information theory allows one to pose problems in principled terms that very often have direct interpretation. For instance, capturing the structure based on statistical regularities of data can be thought of as a problem of relevance determination, that is, information preservation under limited resources. The principle of relevant information is an information theoretic objective function that attempts to capture the statistical regularities through entropy minimization under an information preservation constraint. Here, we employ an information theoretic reproducing kernel Hilbert space (RKHS) formulation, which can overcome some of the limitations of previous approaches based on Parzen density estimation. Results are competitive with kernel-based feature extractors such as kernel PCA. Moreover, the proposed framework goes further on the relation between information theoretic learning, kernel methods and support vector algorithms.