Heter-Sim: Heterogeneous Multi-Agent Systems Simulation by Interactive Data-Driven Optimization

Jiaping Ren, Wei Xiang, Yangxi Xiao, Ruigang Yang, Dinesh Manocha, Xiaogang Jin

Research output: Contribution to journalArticlepeer-review

23 Scopus citations


Interactive multi-agent simulation algorithms are used to compute the trajectories and behaviors of different entities in virtual reality scenarios. However, current methods involve considerable parameter tweaking to generate plausible behaviors. We introduce a novel approach (Heter-Sim) that combines physics-based simulation methods with data-driven techniques using an optimization-based formulation. Our approach is general and can simulate heterogeneous agents corresponding to human crowds, traffic, vehicles, or combinations of different agents with varying dynamics. We estimate motion states from real-world datasets that include information about position, velocity, and control direction. Our optimization algorithm considers several constraints, including velocity continuity, collision avoidance, attraction, direction control. Other constraints are implemented by introducing a novel energy function to control the motions of heterogeneous agents. To accelerate the computations, we reduce the search space for both collision avoidance and optimal solution computation. Heter-Sim can simulate tens or hundreds of agents at interactive rates and we compare its accuracy with real-world datasets and prior algorithms. We also perform user studies that evaluate the plausible behaviors generated by our algorithm and a user study that evaluates the plausibility of our algorithm via VR.

Original languageEnglish
Article number8865441
Pages (from-to)1953-1966
Number of pages14
JournalIEEE Transactions on Visualization and Computer Graphics
Issue number3
StatePublished - Mar 1 2021

Bibliographical note

Funding Information:
Xiaogang Jin was supported by the National Key R&D Program of China (Grant No. 2017YFB1002600), Artificial Intelligence Research Foundation of Baidu Inc., the Key Research and Development Program of Zhejiang Province (Grant No. 2018C01090), and the National Natural Science Foundation of China (Grant No. 61972344). Dinesh Manocha is supported in part by ARO Grant No. W911NF-19-1- 0069 and Intel. Some of the work was done while the first author was an intern at Baidu Research. The authors thank the reviewers for valuable suggestions.


  • Multi-agent model
  • data-driven method
  • heterogeneous group
  • physically driven simulation

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Heter-Sim: Heterogeneous Multi-Agent Systems Simulation by Interactive Data-Driven Optimization'. Together they form a unique fingerprint.

Cite this