CPA-G&V: Self-Completion of 4D (Space+Time)Models

This proposal seeks to address the modeling problem in computer graphk~. The focus is to develop algorithms to facilitate the digitization of our physical world. We envision the use of as few as one single pair of video cameras or one full-frame ranger sensor to capture a dynamic scene over time from different locations. From the sequence of color+depth maps, which provides a partial sampling of the entire 4D (space+time) model, the central research problem we would like to address is how to fuse these partial samples to form a complete model. Different from all previous hole filling approaches, we aim to deal with models that exhibit one or more of the following characteristics: large (e.g., several city blocks), dynamic, deforming, yet sparsely sampled (e.g., no more than 50% is available), and possibly very noisy (e.g., depth maps from passive stereovision). Reconstructing a complete model under these conditions can be very challenging or sometime ill-posed. ilowever, scene structures are usually not stochastic (if they are, examplar-based texture synthesis approaches should work well), the same or similar structure element may have appeared in the input set a few times, probably at a different time and location. The motion is not random either, it is governed by physical laws. The more temporal samples we have, the more likely we can observe the structures and estimate the parameters for the motion trajectory. Therefore, samples from different time or space can be used to fill in the missing data, making model completion possible, without using any external sources. What we are effectively doing is to synthesize the model that is both visually plausible and con- sistent with input data (color+depth). The intellectual merit of this proposal is the development of several different synthesis strategies based on the characteristics of different scene types. First, we develop efficient linear algorithms to simultaneously find all similar structure elements in all images using data-mining techniques. This is particularly useful for scenes with regular structure (such as buildings). Secondly, we introduce the concept of combining image-based modeling with physically based simulation. We will demonstrate its effectiveness on modeling fluid, for example, water fountains, which have not been reconstructed geometrically from images. Thirdly, we develop a novel depth map fusion algorithm based on constrained volumetric warping. It is mainly designed for articulated objects with non-rigid deformation, such as human motion. Both the second and the third approaches will allow the reconstruction of dynamic scenes from a single color+depth sequence, without using a surrounding camera array. The broad impact of this proposal is to simplify the labor intensive process of modeling, thus making 3D contents more accessible. The reconstructed real world 3D models have broad applications in many areas such as simulation and training, education, entertainment, medicine, and archaeology. The ability to create photo-realistic dynamic models from a hand-held stereoscopic camera could significantly change how 3D contents are created, and may eventually lead to the digitalization of our world through a community effort. Maybe in the near future, 3D interactive video will be as pervasive as video is today. Expected results from this proposal are particularly well sulted for out-reach activities. We have established a collaboration with a local high-school to distribute inexpensive DV camcorders with stereoscopic adapters. The goal is to let students create their own 3D contents from videos, inspiring their interests in mathematics and science.