第15卷第3期 智能系统学报 Vol.15 No.3 2020年5月 CAAI Transactions on Intelligent Systems May 2020 D0:10.11992tis.202006029 时空域融合的骨架动作识别与交互研究 钟秋波2，郑彩明，朴松吴 (1.宁波工程学院机器人学院，浙江宁波315211,2.哈尔滨工业大学机器人系统与技术国家重点实验室，黑 龙江哈尔滨150001；3.哈尔滨工业大学计算机科学与技术学院，黑龙江哈尔滨150001) 摘要：在人体骨架结构动作识别方法中，很多研究工作在提取骨架结构上的空间信息和运动信息后进行融 合，没有对具有复杂时空关系的人体动作进行高效表达。本文提出了基于姿态运动时空域融合的图卷积网络 模型(PM-STFGCN)。对于在时域上存在大量的干扰信息，定义了一种基于局部姿态运动的时域关注度模块 (LPM-TAM),用于抑制时域上的干扰并学习运动姿态的表征。设计了基于姿态运动的时空域融合模块(PM ST℉).融合时域运动和空域姿态特征并进行自适应特征增强。通过实验验证，本文提出的方法是有效性的，与 其他方法相比，在识别效果上具有很好的竞争力。设计的人体动作交互系统，验证了在实时性和准确率上优于 语音交互系统。 关键词：动作识别：时空关系：姿态运动：时空域融合；图卷积神经网络：时域关注度；自适应特征增强；人体动 作交互 中图分类号：TP312文献标志码：A文章编号：1673-4785(2020)03-0601-08 中文引用格式：钟秋波，郑彩明，朴松吴.时空域融合的骨架动作识别与交互研究.智能系统学报，2020,15(3)：601-608. 英文引用格式：ZHONG Qiubo,ZHENG Caiming,PIAO Songhao.Research on skeleton-based action recognition with spatiotem poral fusion and human-robot interactionJ CAAI transactions on intelligent systems,2020,15(3):601-608. Research on skeleton-based action recognition with spatiotemporal fusion and human-robot interaction ZHONG Qiubo2,ZHENG Caiming',PIAO Songhao (1.Robotics Institute,Ningbo University of Technology,Ningbo 315211,China;2.State Key Laboratory of Robotics and System, Harbin Institute of Technology,Harbin 150001,China;3.School of Computer Science and Technology,Harbin Institute of Techno- logy,Harbin 150001,China) Abstract:Temporal dynamics of postures over time is crucial for sequence-based action recognition.Human actions can be represented by corresponding motions of an articulated skeleton.Skeleton-based action recognition algorithm is used for studying motions of a body.Skeleton-based action recognition uses many methods,and research shows that most of them extract spatial and motion information separately from a skeleton structure and then combine them for further pro- cessing.However,this process is not able to efficiently deliver human motion features with complex temporal and spa- tial relationships.We propose a novel posture motion-based,spatiotemporal fused graph convolution network for skelet- on-based action recognition.First,we define a local posture motion-based time attention module,which is used to con- strain the disturbance information in temporal domain and learn the representation of motion posture features.Then,we design a posture motion-based,spatiotemporal fusion module.This module fuses spatial motion and temporal attitude features and adaptively enhances the skeleton joint features.Extensive experiments have been performed and the results verified the effectiveness of our proposed method.The proposed method has competitive performance,and it is con- cluded that the human-robot interaction system based on action recognition is superior to the speech interaction system in real-time and with respect to accuracy. Keywords:action recognition;temporal and spatial relationships;posture motion;spatiotemporal fusion; graph convolution network;temporal attention;adaptive feature enhancement;human-robot interaction 收稿日期：2020-06-17. 随着人工智能技术的发展，以人为核心的视 基金项目：国家自然科学基金项目(61203360,61502256)：浙江 觉人机交互技术的关键在于理解人类活动和社 省自然科学基金项目(LQ12F03001). 通信作者：钟秋波.E-mail:zhongqiubo@nbut.edu.cm 会行为。因此，动作识别在人机交互领域具有DOI: 10.11992/tis.202006029 时空域融合的骨架动作识别与交互研究 钟秋波1,2，郑彩明1 ，朴松昊3 （1. 宁波工程学院 机器人学院，浙江 宁波 315211; 2. 哈尔滨工业大学 机器人系统与技术国家重点实验室，黑 龙江 哈尔滨 150001; 3. 哈尔滨工业大学 计算机科学与技术学院，黑龙江 哈尔滨 150001） 摘 要：在人体骨架结构动作识别方法中，很多研究工作在提取骨架结构上的空间信息和运动信息后进行融 合，没有对具有复杂时空关系的人体动作进行高效表达。本文提出了基于姿态运动时空域融合的图卷积网络 模型 (PM-STFGCN)。对于在时域上存在大量的干扰信息，定义了一种基于局部姿态运动的时域关注度模块 (LPM-TAM)，用于抑制时域上的干扰并学习运动姿态的表征。设计了基于姿态运动的时空域融合模块 (PM￾STF)，融合时域运动和空域姿态特征并进行自适应特征增强。通过实验验证，本文提出的方法是有效性的，与 其他方法相比，在识别效果上具有很好的竞争力。设计的人体动作交互系统，验证了在实时性和准确率上优于 语音交互系统。 关键词：动作识别；时空关系；姿态运动；时空域融合；图卷积神经网络；时域关注度；自适应特征增强；人体动 作交互 中图分类号：TP312 文献标志码：A 文章编号：1673−4785(2020)03−0601−08 中文引用格式：钟秋波, 郑彩明, 朴松昊. 时空域融合的骨架动作识别与交互研究 [J]. 智能系统学报, 2020, 15(3): 601–608. 英文引用格式：ZHONG Qiubo, ZHENG Caiming, PIAO Songhao. Research on skeleton-based action recognition with spatiotem￾poral fusion and human–robot interaction[J]. CAAI transactions on intelligent systems, 2020, 15(3): 601–608. Research on skeleton-based action recognition with spatiotemporal fusion and human–robot interaction ZHONG Qiubo1,2 ，ZHENG Caiming1 ，PIAO Songhao3 (1. Robotics Institute, Ningbo University of Technology, Ningbo 315211, China; 2. State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China; 3. School of Computer Science and Technology, Harbin Institute of Techno￾logy, Harbin 150001, China) Abstract: Temporal dynamics of postures over time is crucial for sequence-based action recognition. Human actions can be represented by corresponding motions of an articulated skeleton. Skeleton-based action recognition algorithm is used for studying motions of a body. Skeleton-based action recognition uses many methods, and research shows that most of them extract spatial and motion information separately from a skeleton structure and then combine them for further pro￾cessing. However, this process is not able to efficiently deliver human motion features with complex temporal and spa￾tial relationships. We propose a novel posture motion-based, spatiotemporal fused graph convolution network for skelet￾on-based action recognition. First, we define a local posture motion-based time attention module , which is used to con￾strain the disturbance information in temporal domain and learn the representation of motion posture features. Then, we design a posture motion-based, spatiotemporal fusion module. This module fuses spatial motion and temporal attitude features and adaptively enhances the skeleton joint features. Extensive experiments have been performed and the results verified the effectiveness of our proposed method. The proposed method has competitive performance, and it is con￾cluded that the human–robot interaction system based on action recognition is superior to the speech interaction system in real-time and with respect to accuracy. Keywords: action recognition; temporal and spatial relationships; posture motion; spatiotemporal fusion; graph convolution network; temporal attention; adaptive feature enhancement; human–robot interaction 随着人工智能技术的发展，以人为核心的视 觉人机交互技术的关键在于理解人类活动[1] 和社 会行为[2]。因此，动作识别在人机交互领域具有 收稿日期：2020−06−17. 基金项目：国家自然科学基金项目 (61203360，61502256)；浙江 省自然科学基金项目 (LQ12F03001). 通信作者：钟秋波. E-mail：zhongqiubo@nbut.edu.cn. 第 15 卷第 3 期 智 能 系 统 学 报 Vol.15 No.3 2020 年 5 月 CAAI Transactions on Intelligent Systems May 2020