Human segmentation by geometrically fusing visible-light and thermal imageries

Research output: Contribution to journalArticlepeer-review

17 Scopus citations


From depth sensors to thermal cameras, the increased availability of camera sensors beyond the visible spectrum has created many exciting applications. Most of these applications require combining information from these hyperspectral cameras with a regular RGB camera. Information fusion from multiple heterogeneous cameras can be a very complex problem. They can be fused at different levels from pixel to voxel or even semantic objects, with large variations in accuracy, communication, and computation costs. In this paper, we propose a system for robust segmentation of human figures in video sequences by fusing visible-light and thermal imageries. Our system focuses on the geometric transformation between visual blobs corresponding to human figures observed at both cameras. This approach provides the most reliable fusion at the expense of high computation and communication costs. To reduce the computational complexity of the geometric fusion, an efficient calibration procedure is first applied to rectify the two camera views without the complex procedure of estimating the intrinsic parameters of the cameras. To geometrically register different blobs at the pixel level, a blob-to-blob homography in the rectified domain is then computed in real-time by estimating the disparity for each blob-pair. Precise segmentation is finally achieved using a two-tier tracking algorithm and a unified background model. Our experimental results show that our proposed system provides significant improvements over existing schemes under various conditions.

Original languageEnglish
Pages (from-to)61-89
Number of pages29
JournalMultimedia Tools and Applications
Issue number1
StatePublished - Sep 17 2014

Bibliographical note

Publisher Copyright:
© 2012, Springer Science+Business Media New York.


  • Human segmentation
  • Multi-camera fusion
  • Sensor fusion
  • Thermal cameras

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications


Dive into the research topics of 'Human segmentation by geometrically fusing visible-light and thermal imageries'. Together they form a unique fingerprint.

Cite this