DEPSCoR: Measures of Information via Representation Learning

This work will develop the theoretical foundations of alternative definitions of information theoretic quantities to address a fundamental problem in Machine Learning: building objective functions that can handle and integrate data from multiple sources with minimal supervision. The proposed research will 1) study the properties of the alternative definitions of entropy and mutual information, and establish similarities and differences with their conventional counterparts, 2) describe what properties are relevant when these quantities are used as objective functions, and 3) provide empirical estimators based on representation learning and study their convergence. The proposed approach will be useful across a broad range of domains. These novel information theoretic quantities will be used to develop a robust and scalable approach to multimodal data integration that combines advantages of traditional information theory and modern representation learning based methods.