Human motion recognition and prediction for robot control

Robert X. Gao, Lihui Wang, Peng Wang, Jianjing Zhang, Hongyi Liu

Research output: Chapter in Book/Report/Conference proceedingChapterpeer-review

2 Scopus citations


The ever-increasing demand for higher productivity, lower cost and improved safety continues to drive the advancement of manufacturing technologies. As one of the key elements, human-robot collaboration (HRC) envisions a workspace where humans and robots can dynamically collaborate for improved operational efficiency while maintaining safety. As the effectiveness of HRC is affected by a robot's ability to sense, understand and forecast the state of the collaborating human worker, human action recognition and motion trajectory prediction have become a crucial part in realising HRC. In this chapter, deep-learning-based methods for accomplishing this goal, based on the in-situ sensing data from the workspace are presented. Specifically, to account for the variability and heterogeneity of human workers during assembly, a context-aware deep convolutional neural network (DCNN) has been developed to identify the task-associated context for inferencing human actions. To improve the accuracy and reliability of human motion trajectory prediction, a functional unit-incorporated recurrent neural network (RNN) has been developed to parse worker's motion patterns and forecast worker's future motion trajectories. Collectively, these techniques allow the robot to answer the question: "which tool or part should be delivered to which location next?", and enable online robot action planning and execution for the collaborative assembly operation. The methods developed are experimentally evaluated, with the collaborative assembly of an automotive engine as a case study.

Original languageEnglish
Title of host publicationAdvanced Human-Robot Collaboration in Manufacturing
Number of pages22
ISBN (Electronic)9783030691783
StatePublished - Jun 10 2021

Bibliographical note

Publisher Copyright:
© Springer Nature Switzerland AG 2021. All rights reserved.

ASJC Scopus subject areas

  • General Engineering
  • General Computer Science


Dive into the research topics of 'Human motion recognition and prediction for robot control'. Together they form a unique fingerprint.

Cite this