第17卷第5期 智能系统学报 Vol.17 No.5 2022年9月 CAAI Transactions on Intelligent Systems Sep.2022 D0:10.11992/tis.202107040 网络出版地址：https:/kns.cnki.net/kcms/detail/23.1538.TP.20220519.1429.004html 旋翼无人机在移动平台降落的控制 参数自学习调节方法 张鹏鹏，魏长赞，张恺睿，欧阳勇平 (河海大学机电工程学院，江苏常州213022) 摘要：无人机设备能够适应复杂地形，但由于电池容量等原因，无人机无法长时间执行任务。无人机与其他 无人系统（无人车、无人船等）协同能够有效提升无人机的工作时间，完成既定任务，当无人机完成任务后，将 无人机迅速稳定地降落至移动平台上是一项必要且具有挑战性的工作。针对降落问题，文中提出了基于矫正 纠偏COACH(corrective advice communicated humans)方法的深度强化学习比例积分微分(proportional-integral-de rivative,PID)方法，为无人机降落至移动平台提供了最优路径。首先在仿真环境中使用矫正纠偏框架对强化学 习模型进行训练，然后在仿真环境和真实环境中，使用训练后的模型输出控制参数，最后利用输出参数获得无 人机位置控制量。仿真结果和真实无人机实验表明，基于矫正纠偏COACH方法的深度强化学习PID方法优 于传统控制方法，且能稳定完成在移动平台上的降落任务。 关键词：自主降落；强化学习；路径规划；COACH框架；确定性策略梯度；空地协同：无人机；最优控制 中图分类号：TP273+.2文献标志码：A 文章编号：1673-4785(2022)05-0931-10 中文引用格式：张鹏鹏，魏长簧，张恺容，等.旋翼无人机在移动平台降落的控制参数自学习调节方法.智能系统学报，2022， 17(5):931-940. 英文引用格式：ZHANG Pengpeng,WEI Changyun,,ZHANG Kairui,,etal.Self-learning approach to control parameter adjust- ment for quadcopter landing on a moving platformJ.CAAI transactions on intelligent systems,2022,17(5):931-940. Self-learning approach to control parameter adjustment for quadcopter landing on a moving platform ZHANG Pengpeng,WEI Changyun,ZHANG Kairui,OUYANG Yongping (College of Mechanical and Electrical Engineering,Hohai University,Changzhou 213022,China) Abstract:Unmanned Aerial Vehicle(UAV)is a type of robot that performs well in mapping without being affected by the terrain.However,a UAV cannot perform its tasks for long due to its small battery capacity and several other reasons. The collaboration between UAVs and other unmanned ground vehicles (UGVs)is considered a crucial solution to this concern as it can save up the time taken by UAVs effectively when completing a scheduled task.When deploying a team of UAVs and UGVs,it is both important and challenging to land a UAV on a mobile platform quickly and stably.To circumvent the UAV landing issue,this study proposes a reinforcement learning PID method based on the correction COACH method,thereby providing an optimal path for the UAV to land on a mobile platform.First,the reinforcement learning agent is trained using the rectification framework in a simulated environment.Next,the trained agent is used for output control parameters in the simulated and true environments,and subsequently,the output parameters are utilized to obtain the control variables of the UAV's position.The simulation and real UAV experiment results show that the deep reinforcement learning PID method based on the correction COACH method is superior to the traditional control meth- od and can accomplish the task of a stable landing on a mobile platform. Keywords:autonomous landing;reinforcement learning;path planning;COACH frame;deterministic policy gradient; air-ground cooperation;UAV;optimal control 无人机可以应用于不同的场景，例如日常的 挖掘过程)等。单一的无人机不受地形的限制，但 便民生活应用山，农业生产过程，矿场的侦探和 是由于携带电池能量的限制，执行任务时间短，并 收稿日期：2021-07-20.网络出版日期：2022-05-20 且难以承担较重的负载。无人车（船）移动范围受 基金项目：国家自然科学基金项目(61703138)：中央高校基本 科研业务费项目(B200202224). 限于地形，难以到达特定的位置。无人机-无人车 通信作者：魏长赞.E-mail:c.wei@hhu.edu.cn. (船)的组合系统可以结合两者的优点，完成复杂DOI: 10.11992/tis.202107040 网络出版地址: https://kns.cnki.net/kcms/detail/23.1538.TP.20220519.1429.004.html 旋翼无人机在移动平台降落的控制 参数自学习调节方法 张鹏鹏，魏长赟，张恺睿，欧阳勇平 （河海大学 机电工程学院，江苏 常州 213022） 摘 要：无人机设备能够适应复杂地形，但由于电池容量等原因，无人机无法长时间执行任务。无人机与其他 无人系统（无人车、无人船等）协同能够有效提升无人机的工作时间，完成既定任务，当无人机完成任务后，将 无人机迅速稳定地降落至移动平台上是一项必要且具有挑战性的工作。针对降落问题，文中提出了基于矫正 纠偏 COACH(corrective advice communicated humans) 方法的深度强化学习比例积分微分 (proportional-integral-de￾rivative, PID) 方法，为无人机降落至移动平台提供了最优路径。首先在仿真环境中使用矫正纠偏框架对强化学 习模型进行训练，然后在仿真环境和真实环境中，使用训练后的模型输出控制参数，最后利用输出参数获得无 人机位置控制量。仿真结果和真实无人机实验表明，基于矫正纠偏 COACH 方法的深度强化学习 PID 方法优 于传统控制方法，且能稳定完成在移动平台上的降落任务。 关键词：自主降落；强化学习；路径规划；COACH 框架；确定性策略梯度；空地协同；无人机；最优控制 中图分类号：TP273+.2 文献标志码：A 文章编号：1673−4785(2022)05−0931−10 中文引用格式：张鹏鹏, 魏长赟, 张恺睿, 等. 旋翼无人机在移动平台降落的控制参数自学习调节方法 [J]. 智能系统学报, 2022, 17(5): 931–940. 英文引用格式：ZHANG Pengpeng, WEI Changyun, ZHANG Kairui, et al. Self-learning approach to control parameter adjust￾ment for quadcopter landing on a moving platform[J]. CAAI transactions on intelligent systems, 2022, 17(5): 931–940. Self-learning approach to control parameter adjustment for quadcopter landing on a moving platform ZHANG Pengpeng，WEI Changyun，ZHANG Kairui，OUYANG Yongping (College of Mechanical and Electrical Engineering, Hohai University, Changzhou 213022, China) Abstract: Unmanned Aerial Vehicle (UAV) is a type of robot that performs well in mapping without being affected by the terrain. However, a UAV cannot perform its tasks for long due to its small battery capacity and several other reasons. The collaboration between UAVs and other unmanned ground vehicles (UGVs) is considered a crucial solution to this concern as it can save up the time taken by UAVs effectively when completing a scheduled task. When deploying a team of UAVs and UGVs, it is both important and challenging to land a UAV on a mobile platform quickly and stably. To circumvent the UAV landing issue, this study proposes a reinforcement learning PID method based on the correction COACH method, thereby providing an optimal path for the UAV to land on a mobile platform. First, the reinforcement learning agent is trained using the rectification framework in a simulated environment. Next, the trained agent is used for output control parameters in the simulated and true environments, and subsequently, the output parameters are utilized to obtain the control variables of the UAV’s position. The simulation and real UAV experiment results show that the deep reinforcement learning PID method based on the correction COACH method is superior to the traditional control meth￾od and can accomplish the task of a stable landing on a mobile platform. Keywords: autonomous landing; reinforcement learning; path planning; COACH frame; deterministic policy gradient; air-ground cooperation; UAV; optimal control 无人机可以应用于不同的场景，例如日常的 便民生活应用[1] ，农业生产过程[2] ，矿场的侦探和 挖掘过程[3] 等。单一的无人机不受地形的限制，但 是由于携带电池能量的限制，执行任务时间短，并 且难以承担较重的负载。无人车（船）移动范围受 限于地形，难以到达特定的位置。无人机–无人车 （船）的组合系统可以结合两者的优点，完成复杂 收稿日期：2021−07−20. 网络出版日期：2022−05−20. 基金项目：国家自然科学基金项目（61703138）；中央高校基本 科研业务费项目（B200202224）. 通信作者：魏长赟. E-mail：c.wei@hhu.edu.cn. 第 17 卷第 5 期 智 能 系 统 学 报 Vol.17 No.5 2022 年 9 月 CAAI Transactions on Intelligent Systems Sep. 2022