This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING Multi-Touch in the Air:Concurrent Micromovement Recognition Using RF Signals Lei Xie,Member,IEEE,Chuyu Wang,Student Member,IEEE,Alex X.Liu,Senior Member,IEEE, Jiangiang Sun,and Sanglu Lu,Member,IEEE Abstract-The human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards/touchpads to the brand-new approaches of performing interactions in the air.In this paper,we propose RF-glove,a sys- tem that recognizes concurrent multiple finger micromovement using RF signals,so as to realize the vision of "multi-touch in 1)Zoom In 2)Zoom OUT 3)Rotate Left4)Rotate Right (ZI) (RL) (RR) the air."It uses a commercial-off-the-shelf(COTS)RFID reader with three antennas and five COTS tags attached to the five fingers of a glove,one tag per finger.During the process of a user performing finger micromovements,we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag.For each antenna-tag pair,the reader 5)Flick 6)Swipe Left 7)Swipe Right 8)Punch (SL) (SR) obtains a sequence of RF phase values called a phase profile P門 from the tag's responses over time.To tradeoff between accuracy Fig.1.Example finger micromovements. and robustness in terms of matching resolution,we propose a two phase approach,including coarse-grained filtering and fine- grained matching.To tackle the variation of template phase in a more natural approach,such that the user can directly profiles at different positions,we propose a phase-model-based manipulate the virtual or real objects in the air.This paper con- solution to reconstruct the template phase profiles based on cerns multi-touch in the air,i.e.,the recognition of concurrent the exact locations.Experiment results show that we achieve micromovements using Radio Frequency(RF)signals in RFID an average accuracy of 92.1%under various moving speeds, systems [4]-9.In particular,we consider the concurrent orientation deviations,and so on. micromovements of multiple fingers such as zoom in/out, Index Terms-Passive RFID,RF Signal,micromovement rotate left/right,flick,swipe left/right and punch,as illustrated recognition,prototype design. in Fig.1.This is useful for many applications that requires human-computer interaction through fine-grained concurrent I.INTRODUCTION finger micromovements,such as motion sensing games.For example,a user can manipulate a virtual object with his finger A.Motivation micromovements,such as rotating or stretching the object. TOWADAYS,the human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards and touchpads to the brand-new B.Summary and Limitations of Prior Art approaches of performing interactions in the air.The users can Existing motion sensing technologies use either cameras perform the interactions with the computer using their arms, or sensors.Microsoft Kinect [1]and Leap Motion [3]con- legs and even fingers [1]-[3].In this way,the applications trollers use cameras to capture human motions based on vision of virtual reality and augmented reality can be supported technologies.The key limitation of camera based schemes is that they are more or less affected by the viewing angle Manuscript received December 13.2016:revised July 18.2017:accepted and light condition.Nintendo Wii [2]video game systems November 5,2017;approved by IEEE/ACM TRANSACTIONS ON NETWORK- ING Editor X.Zhou.This work was supported in part by the National Natural use wearable sensors based on infrared technologies.The key Science Foundation of China under Grant 61472185,Grant 61472184,Grant limitation of sensor based schemes is that the sensors are 61373129.Grant 61321491,and Grant 61502224.in part by the Jiangsu Nat- ural Science Foundation under Grant BK20151390,in part by the Fundamental often too big to be conveniently wear.Recently RF-IDraw Research Funds for the Central Universities under Grant 020214380035,in is proposed to use a 2-dimensional array of RFID antennas to part by the National Science Foundation under Grant CNS-1421407,in part by track the movement trajectory of a finger,which is attached the Jiangsu Innovation and Entrepreneurship (Shuangchuang)Program,and with an RFID tag [10].It constructs an efficient beam for in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization.(Corresponding authors:Alex X.Liu:Sanglu Lu.) detecting the finger moving direction by intersecting the beams The authors are with the State Key Laboratory for Novel Software of multiple antennas.However,RF-IDraw is designed to track Technology,Nanjing University.Nanjing 210023,China (e-mail: a fairly large range movement of one finger,e.g.,in the size of Ixie@nju.edu.cn: wangcyu217@dislab.nju.edu.cn; alexliu@nju.edu.cn: sunjq@dislab.nju.edu.cn:sanglu@nju.edu.cn). 20~30cm.It does not work well for tracking the concurrent Digital Object Identifier 10.1109/TNET.2017.2772781 movements of multiple fingers because its median accuracy 1063-66922017 IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE/ACM TRANSACTIONS ON NETWORKING 1 Multi-Touch in the Air: Concurrent Micromovement Recognition Using RF Signals Lei Xie , Member, IEEE, Chuyu Wang, Student Member, IEEE, Alex X. Liu, Senior Member, IEEE, Jianqiang Sun, and Sanglu Lu, Member, IEEE Abstract— The human–computer interactions have moved from the conventional approaches of entering inputs into the keyboards/touchpads to the brand-new approaches of performing interactions in the air. In this paper, we propose RF-glove, a sys￾tem that recognizes concurrent multiple finger micromovement using RF signals, so as to realize the vision of "multi-touch in the air." It uses a commercial-off-the-shelf (COTS) RFID reader with three antennas and five COTS tags attached to the five fingers of a glove, one tag per finger. During the process of a user performing finger micromovements, we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag. For each antenna–tag pair, the reader obtains a sequence of RF phase values called a phase profile from the tag’s responses over time. To tradeoff between accuracy and robustness in terms of matching resolution, we propose a two phase approach, including coarse-grained filtering and fine￾grained matching. To tackle the variation of template phase profiles at different positions, we propose a phase-model-based solution to reconstruct the template phase profiles based on the exact locations. Experiment results show that we achieve an average accuracy of 92.1% under various moving speeds, orientation deviations, and so on. Index Terms— Passive RFID, RF Signal, micromovement recognition, prototype design. I. INTRODUCTION A. Motivation NOWADAYS, the human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards and touchpads to the brand-new approaches of performing interactions in the air. The users can perform the interactions with the computer using their arms, legs and even fingers [1]–[3]. In this way, the applications of virtual reality and augmented reality can be supported Manuscript received December 13, 2016; revised July 18, 2017; accepted November 5, 2017; approved by IEEE/ACM TRANSACTIONS ON NETWORK￾ING Editor X. Zhou. This work was supported in part by the National Natural Science Foundation of China under Grant 61472185, Grant 61472184, Grant 61373129, Grant 61321491, and Grant 61502224, in part by the Jiangsu Nat￾ural Science Foundation under Grant BK20151390, in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035, in part by the National Science Foundation under Grant CNS-1421407, in part by the Jiangsu Innovation and Entrepreneurship (Shuangchuang) Program, and in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization. (Corresponding authors: Alex X. Liu; Sanglu Lu.) The authors are with the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China (e-mail: lxie@nju.edu.cn; wangcyu217@dislab.nju.edu.cn; alexliu@nju.edu.cn; sunjq@dislab.nju.edu.cn; sanglu@nju.edu.cn). Digital Object Identifier 10.1109/TNET.2017.2772781 Fig. 1. Example finger micromovements. in a more natural approach, such that the user can directly manipulate the virtual or real objects in the air. This paper con￾cerns multi-touch in the air, i.e., the recognition of concurrent micromovements using Radio Frequency (RF) signals in RFID systems [4]–[9]. In particular, we consider the concurrent micromovements of multiple fingers such as zoom in/out, rotate left/right, flick, swipe left/right and punch, as illustrated in Fig. 1. This is useful for many applications that requires human-computer interaction through fine-grained concurrent finger micromovements, such as motion sensing games. For example, a user can manipulate a virtual object with his finger micromovements, such as rotating or stretching the object. B. Summary and Limitations of Prior Art Existing motion sensing technologies use either cameras or sensors. Microsoft Kinect [1] and Leap Motion [3] con￾trollers use cameras to capture human motions based on vision technologies. The key limitation of camera based schemes is that they are more or less affected by the viewing angle and light condition. Nintendo Wii [2] video game systems use wearable sensors based on infrared technologies. The key limitation of sensor based schemes is that the sensors are often too big to be conveniently wear. Recently RF-IDraw is proposed to use a 2-dimensional array of RFID antennas to track the movement trajectory of a finger, which is attached with an RFID tag [10]. It constructs an efficient beam for detecting the finger moving direction by intersecting the beams of multiple antennas. However, RF-IDraw is designed to track a fairly large range movement of one finger, e.g., in the size of 20∼30cm. It does not work well for tracking the concurrent movements of multiple fingers because its median accuracy 1063-6692 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information