Human motion recognition and prediction for robot control

The ever-increasing demand for higher productivity, lower cost and improved safety continues to drive the advancement of manufacturing technologies. As one of the key elements, human-robot collaboration (HRC) envisions a workspace where humans and robots can dynamically collaborate for improved operational efficiency while maintaining safety. As the effectiveness of HRC is affected by a robot's ability to sense, understand and forecast the state of the collaborating human worker, human action recognition and motion trajectory prediction have become a crucial part in realising HRC. In this chapter, deep-learning-based methods for accomplishing this goal, based on the in-situ sensing data from the workspace are presented. Specifically, to account for the variability and heterogeneity of human workers during assembly, a context-aware deep convolutional neural network (DCNN) has been developed to identify the task-associated context for inferencing human actions. To improve the accuracy and reliability of human motion trajectory prediction, a functional unit-incorporated recurrent neural network (RNN) has been developed to parse worker's motion patterns and forecast worker's future motion trajectories. Collectively, these techniques allow the robot to answer the question: "which tool or part should be delivered to which location next?", and enable online robot action planning and execution for the collaborative assembly operation. The methods developed are experimentally evaluated, with the collaborative assembly of an automotive engine as a case study.