Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller

Authors

Jianwen Li, Purdue University
Jalil Chavez-Galaviz, Purdue University
Kamyar Azizzadenesheli, Nvidia Corporation
Nina Mahmoudian, Purdue University

Recommended Citation

Li, J.; Chavez-Galaviz, J.; Azizzadenesheli, K.; Mahmoudian, N. Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller. Sensors 2023, 23, 3572. https://doi.org/10.3390/s23073572

DOI

10.3390/s23073572

Date of this Version

3-29-2023

Keywords

Unmanned surface vehicle; deep reinforcement learning; collision avoidance; model predictive control

Abstract

This work presents a framework that allows Unmanned Surface Vehicles (USVs) to avoid dynamic obstacles through initial training on an Unmanned Ground Vehicle (UGV) and cross-domain retraining on a USV. This is achieved by integrating a Deep Reinforcement Learning (DRL) agent that generates high-level control commands and leveraging a neural network based model predictive controller (NN-MPC) to reach target waypoints and reject disturbances. A Deep Q Network (DQN) utilized in this framework is trained in a ground environment using a Turtlebot robot and retrained in a water environment using the BREAM USV in the Gazebo simulator to avoid dynamic obstacles. The network is then validated in both simulation and real-world tests. The cross-domain learning largely decreases the training time (28%" role="presentation" style="box-sizing: border-box; max-height: none; display: inline; line-height: normal; font-size: 13.2px; overflow-wrap: normal; text-wrap: nowrap; float: none; direction: ltr; max-width: none; min-width: 0px; min-height: 0px; border: 0px; padding: 0px; margin: 0px; color: rgb(34, 34, 34); font-family: Arial, Arial, Helvetica, sans-serif; position: relative;">28%28%) and increases the obstacle avoidance performance (70 more reward points) compared to pure water domain training. This methodology shows that it is possible to leverage the data-rich and accessible ground environments to train DRL agent in data-poor and difficult-to-access marine environments. This will allow rapid and iterative agent development without further training due to the change in environment or vehicle dynamics.

Comments

This is the published version of the Li, J.; Chavez-Galaviz, J.; Azizzadenesheli, K.; Mahmoudian, N. Dynamic Obstacle Avoidance for USVs Using Cross-Domain Deep Reinforcement Learning and Neural Network Model Predictive Controller. Sensors 2023, 23, 3572. https://doi.org/10.3390/s23073572