第17卷第5期 智能系统学报 Vol.17 No.5 2022年9月 CAAI Transactions on Intelligent Systems Sep.2022 D0:10.11992/tis.202107059 网络出版地址：https:/ns.cnki.net/kcms/detail/23.1538.TP.20220623.1044.004html 基于深度强化学习的室内视觉局部路径规划 朱少凯，孟庆浩，金晟，戴旭阳 (天津大学电气自动化与信息工程学院机器人与自主系统研究所，天津300072) 摘要：传统的机器人局部路径规划方法多为已有先验地图的情况设计，导致其在与视觉(simultaneous localiza tion and mapping,SLAM)结合的导航中效果不佳。为此传统的机器人局部路径规划方法多为已有先验地图的情 况设计，导致其在与视觉SLAM结合的导航中效果不佳。为此，本文提出一种基于深度强化学习的视觉局部路 径规划策略。首先，基于视觉同时定位与建图(SLAM)技术建立周围环境的栅格地图，并使用A*算法规划全局 路径；其次，综合考虑避障、机器人行走效率、位姿跟踪等问题，构建基于深度强化学习的局部路径规划策略， 设计以前进、左转、右转为基本元素的离散动作空间.以及基于彩色图、深度图、特征点图等视觉观测的状态空 间，利用近端策略优化(proximal policy optimization,PPO)算法学习和探索最佳状态动作映射网络。Habitat仿真 平台运行结果表明，所提出的局部路径规划策略能够在实时创建的地图上规划出一条最优或次优路径。相比 于传统的局部路径规划算法，平均成功率提高了53.9%，位姿跟踪丢失率减小了66.5%，碰撞率减小了30.1%。 关键词：视觉导航；深度学习；强化学习；局部路径规划；避障；视觉SLAM;近端策略优化：移动机器人 中图分类号：TP391文献标志码：A文章编号：1673-4785(2022)05-0908-11 中文引用格式：朱少凯，孟庆浩，金晟，等.基于深度强化学习的室内视觉局部路径规划.智能系统学报，2022,17(5)： 908-918. 英文引用格式：ZHU Shaokai,,MENG Qinghao,JIN Sheng,etal.Indoor visual local path planning based on deep reinforcement learning[J].CAAI transactions on intelligent systems,2022,17(5):908-918. Indoor visual local path planning based on deep reinforcement learning ZHU Shaokai,MENG Qinghao,JIN Sheng,DAI Xuyang (Institute of Robotics and Autonomous Systems,School of Electrical and Information Engineering,Tianjin University,300072, China) Abstract:Traditional robot local path planning methods are mostly designed for situations with prior maps,thus lead- ing to poor results in navigation when combined with visual simultaneous localization and mapping(SLAM).Therefore, this paper proposes a visual local path planning strategy based on deep reinforcement learning.First,a grid map of the surrounding environment is built based on the visual SLAM technology,and the global path is planned using the A*al- gorithm.Second,considering the problems of obstacle avoidance,robot walking efficiency,and pose tracking,a local path planning strategy is constructed based on deep reinforcement learning to design the discrete action space with for- ward movement,left turn,and right turn as the basic elements,as well as the state space based on visual observation maps,such as color,depth,and feature point maps.The proximal policy optimization(PPO)algorithm is used to learn and explore the best state-action mapping network.The running results of the habitat simulation platform show that the proposed local path planning strategy can design an optimal or sub-optimal path on a map generated in real time.Com- pared with traditional local path planning algorithms,the average success rate of the proposed strategy is increased by 53.9%,and the average tracking failure rate and collision rate are reduced by 66.5%and 30.1%,respectively. Keywords:visual navigation:deep learning:reinforcement learning:local path planning;obstacle avoidance;visual SLAM;proximal policy optimization(PPO);mobile robot 收稿日期：2021-07-27.网络出版日期：2022-06-24 视觉导航是一类新兴的导航技术，具有使用 基金项目：中国博士后科学基金项目(2021M692390):天津市自 成本低、获取信息丰富的优点，成为近些年来机 然科学基金项目(20 JCZDJC00150,20 JCYBJC00320). 通信作者：金最.E-mail:shengjin@ju.edu.cn. 器人领域的研究热点之一。路径规划是移动DOI: 10.11992/tis.202107059 网络出版地址: https://kns.cnki.net/kcms/detail/23.1538.TP.20220623.1044.004.html 基于深度强化学习的室内视觉局部路径规划 朱少凯，孟庆浩，金晟，戴旭阳 （天津大学 电气自动化与信息工程学院 机器人与自主系统研究所，天津 300072） 摘 要：传统的机器人局部路径规划方法多为已有先验地图的情况设计，导致其在与视觉 (simultaneous localiza￾tion and mapping, SLAM) 结合的导航中效果不佳。为此传统的机器人局部路径规划方法多为已有先验地图的情 况设计，导致其在与视觉 SLAM 结合的导航中效果不佳。为此，本文提出一种基于深度强化学习的视觉局部路 径规划策略。首先，基于视觉同时定位与建图 (SLAM) 技术建立周围环境的栅格地图，并使用 A*算法规划全局 路径；其次，综合考虑避障、机器人行走效率、位姿跟踪等问题，构建基于深度强化学习的局部路径规划策略， 设计以前进、左转、右转为基本元素的离散动作空间，以及基于彩色图、深度图、特征点图等视觉观测的状态空 间，利用近端策略优化 (proximal policy optimization, PPO) 算法学习和探索最佳状态动作映射网络。Habitat 仿真 平台运行结果表明，所提出的局部路径规划策略能够在实时创建的地图上规划出一条最优或次优路径。相比 于传统的局部路径规划算法，平均成功率提高了 53.9%，位姿跟踪丢失率减小了 66.5%，碰撞率减小了 30.1%。 关键词：视觉导航；深度学习；强化学习；局部路径规划；避障；视觉 SLAM；近端策略优化；移动机器人 中图分类号：TP391 文献标志码：A 文章编号：1673−4785(2022)05−0908−11 中文引用格式：朱少凯, 孟庆浩, 金晟, 等. 基于深度强化学习的室内视觉局部路径规划 [J]. 智能系统学报, 2022, 17(5): 908–918. 英文引用格式：ZHU Shaokai, MENG Qinghao, JIN Sheng, et al. Indoor visual local path planning based on deep reinforcement learning[J]. CAAI transactions on intelligent systems, 2022, 17(5): 908–918. Indoor visual local path planning based on deep reinforcement learning ZHU Shaokai，MENG Qinghao，JIN Sheng，DAI Xuyang (Institute of Robotics and Autonomous Systems, School of Electrical and Information Engineering, Tianjin University, 300072, China) Abstract: Traditional robot local path planning methods are mostly designed for situations with prior maps, thus lead￾ing to poor results in navigation when combined with visual simultaneous localization and mapping (SLAM). Therefore, this paper proposes a visual local path planning strategy based on deep reinforcement learning. First, a grid map of the surrounding environment is built based on the visual SLAM technology, and the global path is planned using the A* al￾gorithm. Second, considering the problems of obstacle avoidance, robot walking efficiency, and pose tracking, a local path planning strategy is constructed based on deep reinforcement learning to design the discrete action space with for￾ward movement, left turn, and right turn as the basic elements, as well as the state space based on visual observation maps, such as color, depth, and feature point maps. The proximal policy optimization (PPO) algorithm is used to learn and explore the best state–action mapping network. The running results of the habitat simulation platform show that the proposed local path planning strategy can design an optimal or sub-optimal path on a map generated in real time. Com￾pared with traditional local path planning algorithms, the average success rate of the proposed strategy is increased by 53.9%, and the average tracking failure rate and collision rate are reduced by 66.5% and 30.1%, respectively. Keywords: visual navigation; deep learning; reinforcement learning; local path planning; obstacle avoidance; visual SLAM; proximal policy optimization (PPO); mobile robot 视觉导航是一类新兴的导航技术，具有使用 成本低、获取信息丰富的优点，成为近些年来机 器人领域的研究热点之一[1-4]。路径规划是移动 收稿日期：2021−07−27. 网络出版日期：2022−06−24. 基金项目：中国博士后科学基金项目 (2021M692390)；天津市自 然科学基金项目 (20JCZDJC00150, 20JCYBJC00320). 通信作者：金晟. E-mail：shengjin@tju.edu.cn. 第 17 卷第 5 期 智 能 系 统 学 报 Vol.17 No.5 2022 年 9 月 CAAI Transactions on Intelligent Systems Sep. 2022