Interactive 3D teleconferencing with user-adaptive views

We present a system and techniques for synthesizing views for three-dimensional video teleconferencing. Instead of performing complex 3D scene acquisition, we decided to trade storage/hardware for computation, i.e., using more cameras. While it is expensive to directly capture a scene from all possible viewpoints, we observed that the participants' viewpoints usually remain at a constant height (eye level) during video teleconferencing. Therefore we can restrict the possible viewpoint to be within a virtual plane without sacrificing much of the realism. Doing so significantly reduces the number of cameras required. We demonstrate a real-time system that uses a linear array of cameras to perform Light-Field style rendering. The simplicity and robustness of light fielding rendering, combined with the natural restrictions of limited view volume in video teleconferencing, allow us to synthesize photo-realistic views per user request at interactive rate.