High-quality stereo video matching via user interaction and space-time propagation

Even current state-of-the-art automatic stereo matching methods often struggle on natural images and videos, in great part due to fundamental matching ambiguities in low texture regions and a lack of higher level object knowledge. Stereo image matching can benefit greatly from user input to guide the matching process and help disambiguate matches. Applying interactive correction tools from scratch on each frame of a video would not only be throwing away valuable information provided by the user on other frames, but would also likely be too time consuming to be practical for video even if excellent disparity results could be obtained within a few minutes on each frame. In this work, we propose a stereo video matching system that allows user interaction to obtain high quality, dense disparity maps on key frames and then intelligently propagates the user input and key frame disparities to automatically produce high quality disparity maps on intermediate frames. The disparity maps on key frames are obtained using several novel, easy-to-use, and effective interactive tools. Our novel propagation algorithm estimates 3D transformations that map user corrected areas in key frames to intermediate frames. Experiments demonstrate the effectiveness and efficiency of our hybrid interactive/automatic approach.