CAREER: The Light Portal: 3D Reconstruction and Visualization over Space and Time

Abstract This proposal aims at the problems of reconstruction and visualization of three-dimensional objects from images or video streams. The ultimate goal is an end-to-end light porial that can capture and "re-enact" real-world objects or events in a different time or space. A light portal captures a scene using a practical number of cameras from different viewpoints. From these input images, intelligent software can either reconstruct a coherent parametric scene model that includes not only geometry but also lighting and surface reflectance properties (suitable for storage editing, and playback over time); or directly interpolate the complete bundle of color rays to be displayed (suitable for live transmission over space). These results will be visualized on a unique light-field display that utilizes an array of projectors and microlens screens to emit a large collection of controllable light rays, as if the light rays in the original scene are "tele-ported". Therefore, any number of viewers can simultaneously perceive view-dependent stereoscopic effect as if they were in the real scene. The realization of the light portal poses many scientific and engineering challenges in vision, graphics, image processing, networking, human-computer interaction, optics, etc. This proposal, drawn upon the success of PI's past research, focuses on the core scientific problems in 3D reconstruction and visualization. Intellectual Merits On the reconstruction front, two paths will be explored. One is aimed to allow robust 3D reconstruction for a much broader class of scene types by finding the invariance from the complex interactions among geometry, lighting and surface reflectance properties. Alternatively an inverse formulation to estimate directly the final image will be developed. By incorporating new constraints that are difficult to adopt in a geometric setting, such as nature image statistics, this inverse formulation allows more realistic image synthesis even when explicitly reconstructing the 3D model is ill-conditioned. These two approaches are complimentary to each other with one optimized for storage and editing while the other for direct transmission and visualization. On the visualization front, the proposed light-field display will have a unique design that decouples regular 2D displays and 3D screens, made possible by novel calibration and rendering algorithms. Therefore, in addition to its autostereoscopic capability, it offers an unprecedented level of flexibility and scalability in terms of resolution, size, field of view, etc. Broad Impact Analogous to the ubiquitous digital cameras and video displays in use today, a light portal can provide the ultimate three-dimensional recording and viewing experience that may lead to profound changes in the way people communicate, educate, and entertain. In particular the 3D capability may enable new paradigms for surgical training and tele-collaboration. This project also has an integral educational component. Besides the usual course development, student participation, and broad dissemination through various venues (publication, demonstration, web site, etc.), the PI will make efforts to attract minority (Hispanic) students by taking an active part in the exchange program between the University of Kentucky and the University of Puerto Rico.