Safe Navigation with Human Instructions in Complex Scenes

Zhe Hu, Jia Pan, Tingxiang Fan, Ruigang Yang, Dinesh Manocha

Research output: Contribution to journalArticlepeer-review

19 Scopus citations


In this letter, we present a robotic navigation algorithm with natural language interfaces that enables a robot to safely walk through a changing environment with moving persons by following human instructions such as 'go to the restaurant and keep away from people.' We first classify human instructions into three types: goal, constraints, and uninformative phrases. Next, we provide grounding in a dynamic manner for the extracted goal and constraint items along with the navigation process to deal with target objects that are too far away for sensor observation and the appearance of moving obstacles such as humans. In particular, for a goal phrase (e.g., 'go to the restaurant'), we ground it to a location in a predefined semantic map and treat it as a goal for a global motion planner, which plans a collision-free path in the workspace for the robot to follow. For a constraint phrase (e.g., 'keep away from people'), we dynamically add the corresponding constraint into a local planner by adjusting the values of a local costmap according to the results returned by the object detection module. The updated costmap is then used to compute a local collision avoidance control for the safe navigation of the robot. By combining natural language processing, motion planning, and computer vision, our developed system can successfully follow natural language navigation instructions to achieve navigation tasks in both simulated and real-world scenarios. Videos are available at

Original languageEnglish
Article number8613882
Pages (from-to)753-760
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number2
StatePublished - Apr 2019

Bibliographical note

Publisher Copyright:
© 2016 IEEE.


  • Social human-robot interaction
  • motion and path planning
  • robot audition
  • semantic scene understanding
  • visual-based navigation

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence


Dive into the research topics of 'Safe Navigation with Human Instructions in Complex Scenes'. Together they form a unique fingerprint.

Cite this