III: Small:FoLIO-Framework for Longitudinal Image-based Organization

Project Summary This proposed project will develop a framework for organizing images that allows the specific types of relationships between those images to be represented, manipulated, highlighted, enhanced, and studied. The technical challenges involve building new representational and algorithmic systems to capture the major "longitudinal categories" that relate heterogeneous images to each other within collections. The problem of relationship between images is normally posed through registration, which is most often highly contextualized. This work will capture the steps necessary to specify registration as a metadata construction that enables a range of granularities in mapping images to each other, and heterogeneous relationship across organizational categories such as time (diachronic), multi-modal, and instances related by a semantic object. The work is interdisciplinary, with the PI and Co-PI working together to apply the results in the field of humanities to a large digitized collection of signature manuscripts, starting with the Homeric Iliad. This study will focus on four things: major longitudinal categories that relate images; how to make different kinds of measurement possible through registration; ways to expand the granularity of the range of potential registrations; and definitions of metadata constructs that can support longitudinally-organized image collections in order for automated and semi-automated tools to emerge. There are technical barriers to building longitudinal collections of images and models. During the work these problems will be formalized and the deployed prototype will offer an example of systematically designed solutions. Image recognition and scene understanding, image registration, and image similarity metrics playa role, although this work will focus on using existing methods to study three important categories for how images in a collection relate to one another: multi-modal, diachronic, and multi-instance. Multi-modal imagery is a collection of images of essentially the same scene taken under varying conditions. Diachronic imagery is an image set taken of essentially the same scene but over a wide span of time, such as the images of a building or landmark. Multiinstance imagery is an image set that is connected semantically, like images of the pages of a story written by two completely different scribes (same story, different instances). Intellectual Merit: The challenge of organizing massive, heterogeneous image collections is on the frontier of information integration. The methods we propose can help to capture the power of a collection through relationships, variations, and the need to quickly use data for analysis. There are many open problems with respect to organization within image collections and this work will address important aspects that will be applied to a prominent interdisciplinary collection of manuscript and textual data.