Real-time view synthesis using commodity graphics hardware

We present a novel use of commodity graphics hardware that effectively combines a plane-sweeping algorithm [Collins 1996] and view synthesis in a single step for real-time, on-line 3D view synthesis. Unlike typical stereo algorithms that use image-based metrics to estimate depths, we focus on using image-based metrics to directly estimate images. Using real-time imagery from a few calibrated cameras, our method can generate new images from nearby viewpoints, without any prior geometric information or requiring any user interaction, in real time and on line. For a desired new view C_n, we discretize the 3D space into a number of candidate focal planes {D_i} parallel to the image plane Cn of the desired view. These candidate planes discretize each desired view ray into a finite set of sample points. We step through the set of candidate focal planes, looking for the sample point along each view ray that offers the maximum color consistency among input images. Each final image pixel is determined by the color at this point of maximum consistency. To accomplish this, for each candidate plane Di we project (texture) the input images onto that plane. We then render the resulting textured plane onto the image plane of C_n to get an image (I_i) of D_i. We combine these two operations into a single homography (planar-to-planar) transformation. In the first row of Figure 1, we show a number of images from different planes. Note that each of these images contains the projections from all input images, and the area corresponding to the intersection of objects and the correct candidate focal plane remains sharp. For each pixel location (u,v) in Ii, we compute the mean and Sum of Squared Difference (SSD) score. The final color of (u,v) is the color with minimum SSD score in {I_i}. We have discovered a novel use of graphics hardware to carry out the entire computation on the graphics board. Modern graphic cards, such as NVIDIA's GeForce series, provide a programable means for per-pixel fragment coloring through the use of register combiners [Kilgard 2000]. We exploit this programmability, together with the texture mapping functions, to carry out the entire computation on the graphics board.