Enhanced human-robot collaboration through deep learning enabled human motion prediction

Abstract

Through enhanced sensing and communication, human-robot collaboration (HRC) refers to application scenarios where a robot, usually a collaborative robot (cobot), and a human occupy the same workspace and interact to accomplish collaborative tasks. Efficiency and safety are among the most important considerations in these scenarios.

Toward this target, we first consider the perception of a HRC system, focusing on the development of a that enables the robot to understand the task and `co-workers' behavior using human action recognition and motion prediction technologies. Using these technologies the current task category and the future poses of the human can be estimated based on observations of the human's recent history of movement patterns and actions. In this research, these technologies are employed to support the early detection of human intention to improve the efficiency of HRC by minimizing the idle time of both the robot and the human in HRC tasks. Further, the prediction horizon is extended by solving the long-term dependency problem using deep neural networks (DNNs) to provide more information to the robot based on unfinished human behavior. The results prove that the designed perception model is able to provide recognition and prediction with higher accuracy than existing models.

Subsequently, a controller that interacts with the perception system is designed to enable robots to replan and modify the current trajectory in real time. To enable safe and effective human–robot collaboration in smart manufacturing, seamless integration of sensing, cognition, and prediction into the robot controller is critical for real-time awareness, response, and communication inside a heterogeneous environment (robots, humans, and equipment). To achieve this, a novel control methodology is proposed that takes advantage of the prediction capabilities of nonlinear model predictive control (NMPC) to execute path-planning tasks based on feedback from a vision system. To further guarantee the safety of humans, a novel safety-critical paradigm using a control barrier function (CBF) is incorporated within the NMPC framework as a safety filter. This is successfully applied to a 7 degrees-of-freedom (DOFs) manipulator in three-dimensional space. Results for a screwdriver handover case study show that the designed controller is able to collaborate with humans safely and smoothly, while improving the efficiency of HRC by reducing idle time by 19.8\%.

To further guarantee the safety of human operators in HRC, a safety-critical control (SCC) based controller, is proposed to implement a HRC task involving path tracking and collision avoidance. In the real world, the uncertainty of the system leads to the potential for accidents during HRC. To address this we will use radial basis function neural networks (RBFNN) to estimate the unknown terms in the controller. Additionally, the idea of the integral barrier Lyapunov function (IBLF) is employed to enable the robot to work within a constrained area for safety purposes.

Apart from the safety consideration, improving the performance of collision avoidance is challenging. To address this problem we develop a quadratic penalty programming method to achieve a collision-free and path-saving trajectory in HRC. Compared with conventional control barrier function (CBF) - based methods, this work is believed to be the first to use a ML-based penalty term to regulate the robot's motion as it approaches the safe boundary. It is also the first to propagate the system uncertainty into multiple SCC tools. Simulations on a redundant manipulator demonstrate that the proposed approach greatly improves the performance of path tracking and collision avoidance, while guaranteeing the safety of human co-workers.

Date of Award	Jul 2023
Original language	English
Awarding Institution	Queen's University Belfast
Supervisor	Seán McLoone (Supervisor) & Mien Van (Supervisor)