第16卷第4期 智能系统学报 Vol.16 No.4 2021年7月 CAAI Transactions on Intelligent Systems Jul.2021 D0:10.11992/tis.202007042 网络出版地址：https:/kns.cnki.net/kcms/detail/23.1538.TP.20210507.1642.002.html 基于反馈注意力机制和上下文融合的非模式实例分割 董俊杰'，刘华平2，谢琚，续欣莹，孙富春 (1.太原理工大学信息与计算机学院，山西晋中030600：2.清华大学智能技术与系统国家重点实验室，北京 100084,3.太原理工大学电气与动力工程学院，山西太原030024) 摘要：非模式实例分割是最近提出的对实例分割的扩展，其任务是对每个对象实例的可见区域和被遮挡区域 都进行预测，感知完整的物理结构和语义概念。在预测对象被遮挡部分的形状和语义时，往往由于特征表示的 识别能力不够和对上下文信息缺乏而导致对遮挡区域预测欠拟合甚至错误。针对这个问题，提出一个上下文 注意模块和反馈注意力机制的特征金字塔结构，引人反馈连接进行再学习。该方法能够有效捕获全局语义信 息和精细的空间细节，通过在COCO-amodal数据集训练和验证，非模式实例分割掩码平均精确率从8.4%提高 到14.3%，平均召回率从16.6%提高到20.8%。实验结果表明，该方法能够显著提高对物体被遮挡部分预测的 准确率，有效解决欠拟合问题。 关键词：非模式实例分割：遮挡预测：反馈连接：注意力机制：上下文信息：深度学习：神经网络：计算机视觉 中图分类号：TP183文献标志码：A文章编号：1673-4785(2021)04-0801-10 中文引用格式：董俊杰，刘华平，谢裙，等.基于反馈注意力机制和上下文融合的非模式实例分割J.智能系统学报，2021， 16(4):801-810. 英文引用格式：DONGJunjie,LIU Huaping,XIEJun,,etal.Feedback attention mechanism and context fusion based amodal in- stance segmentation[J].CAAI transactions on intelligent systems,2021,16(4):801-810. Feedback attention mechanism and context fusion based amodal instance segmentation DONG Junjie',LIU Huaping,XIE Jun',XU Xinying,SUN Fuchun (1.College of Information and Computer,Taiyuan University of Technology,Jinzhong 030600,China;2.State Key Lab.of Intelli- gent Technology and Systems,Tsinghua University,Beijing 100084,China;3.College of Electrical and Power Engineering,Taiyuan University of Technology,Taiyuan 030024,China) Abstract:Recently,model instance segmentation has been proposed as an extension of instance segmentation to predict the visible and occluded areas of each object instance and perceive the complete physical structure and semantic con- cepts.When the shapes and meanings of occluded objects are being predicted,underfitting or even wrong results are ob- tained in the occlusion prediction due to the insufficient recognition capability of feature representation and the lack of contextual information.To solve this problem,this paper proposes a contextual attention module and feature pyramid structure of feedback attention mechanism and introduces feedback connections for relearning.The proposed method can effectively capture global semantic information and fine spatial details.Through training and verification in the COCO-amodal dataset,the average precision of the amodal instance segmentation mask increases from 8.4%to 14.3%, and the average recall rate increases from 16.6%to 20.8%.Experimental results show that this method can significantly improve the accuracy of occlusion prediction and effectively end underfitting. Keywords:amodal instance segmentation;occlusion prediction;feedback connection;attention mechanism;context in- formation;deep learning;neural network;computer vision 收稿日期：2020-07-24.网络出版日期：2021-05-07. 基金项目：山西省自然科学基金项目(201801D121144, 近年来，图像分类、目标检测361、语义分 201801D221190):辽宁省科技厅机器人技术国家重 点实验室联合基金项目(2020-KF-22-06). 割7-、实例分割90等视觉识别任务取得了巨大 通信作者：刘华平.E-mail:hpliu@tsinghua.edu.cn 的进展。计算机视觉系统的性能在精度上越来越DOI: 10.11992/tis.202007042 网络出版地址: https://kns.cnki.net/kcms/detail/23.1538.TP.20210507.1642.002.html 基于反馈注意力机制和上下文融合的非模式实例分割 董俊杰1 ，刘华平2 ，谢珺1 ，续欣莹3 ，孙富春2 （1. 太原理工大学 信息与计算机学院，山西 晋中 030600; 2. 清华大学 智能技术与系统国家重点实验室，北京 100084; 3. 太原理工大学 电气与动力工程学院，山西 太原 030024） 摘 要：非模式实例分割是最近提出的对实例分割的扩展，其任务是对每个对象实例的可见区域和被遮挡区域 都进行预测，感知完整的物理结构和语义概念。在预测对象被遮挡部分的形状和语义时，往往由于特征表示的 识别能力不够和对上下文信息缺乏而导致对遮挡区域预测欠拟合甚至错误。针对这个问题，提出一个上下文 注意模块和反馈注意力机制的特征金字塔结构，引入反馈连接进行再学习。该方法能够有效捕获全局语义信 息和精细的空间细节，通过在 COCO-amodal 数据集训练和验证，非模式实例分割掩码平均精确率从 8.4% 提高 到 14.3%，平均召回率从 16.6% 提高到 20.8%。实验结果表明，该方法能够显著提高对物体被遮挡部分预测的 准确率，有效解决欠拟合问题。 关键词：非模式实例分割；遮挡预测；反馈连接；注意力机制；上下文信息；深度学习；神经网络；计算机视觉 中图分类号：TP183 文献标志码：A 文章编号：1673−4785(2021)04−0801−10 中文引用格式：董俊杰, 刘华平, 谢珺, 等. 基于反馈注意力机制和上下文融合的非模式实例分割 [J]. 智能系统学报, 2021, 16(4): 801–810. 英文引用格式：DONG Junjie, LIU Huaping, XIE Jun, et al. Feedback attention mechanism and context fusion based amodal in￾stance segmentation[J]. CAAI transactions on intelligent systems, 2021, 16(4): 801–810. Feedback attention mechanism and context fusion based amodal instance segmentation DONG Junjie1 ，LIU Huaping2 ，XIE Jun1 ，XU Xinying3 ，SUN Fuchun2 (1. College of Information and Computer, Taiyuan University of Technology, Jinzhong 030600, China; 2. State Key Lab. of Intelli￾gent Technology and Systems, Tsinghua University, Beijing 100084, China; 3. College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China) Abstract: Recently, model instance segmentation has been proposed as an extension of instance segmentation to predict the visible and occluded areas of each object instance and perceive the complete physical structure and semantic con￾cepts. When the shapes and meanings of occluded objects are being predicted, underfitting or even wrong results are ob￾tained in the occlusion prediction due to the insufficient recognition capability of feature representation and the lack of contextual information. To solve this problem, this paper proposes a contextual attention module and feature pyramid structure of feedback attention mechanism and introduces feedback connections for relearning. The proposed method can effectively capture global semantic information and fine spatial details. Through training and verification in the COCO-amodal dataset, the average precision of the amodal instance segmentation mask increases from 8.4% to 14.3%, and the average recall rate increases from 16.6% to 20.8%. Experimental results show that this method can significantly improve the accuracy of occlusion prediction and effectively end underfitting. Keywords: amodal instance segmentation; occlusion prediction; feedback connection; attention mechanism; context in￾formation; deep learning; neural network; computer vision 近年来，图像分类[1-2] 、目标检测[3-6] 、语义分 割 [7-8] 、实例分割[9-10] 等视觉识别任务取得了巨大 的进展。计算机视觉系统的性能在精度上越来越 收稿日期：2020−07−24. 网络出版日期：2021−05−07. 基金项目：山西省自然科学基金项 目 (201801D121144 ， 201801D221190)；辽宁省科技厅机器人技术国家重 点实验室联合基金项目 (2020-KF-22-06). 通信作者：刘华平. E-mail：hpliu@tsinghua.edu.cn. 第 16 卷第 4 期 智 能 系 统 学 报 Vol.16 No.4 2021 年 7 月 CAAI Transactions on Intelligent Systems Jul. 2021