This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING Multi-Touch in the Air:Concurrent Micromovement Recognition Using RF Signals Lei Xie,Member,IEEE,Chuyu Wang,Student Member,IEEE,Alex X.Liu,Senior Member,IEEE, Jiangiang Sun,and Sanglu Lu,Member,IEEE Abstract-The human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards/touchpads to the brand-new approaches of performing interactions in the air.In this paper,we propose RF-glove,a sys- tem that recognizes concurrent multiple finger micromovement using RF signals,so as to realize the vision of "multi-touch in 1)Zoom In 2)Zoom OUT 3)Rotate Left4)Rotate Right (ZI) (RL) (RR) the air."It uses a commercial-off-the-shelf(COTS)RFID reader with three antennas and five COTS tags attached to the five fingers of a glove,one tag per finger.During the process of a user performing finger micromovements,we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag.For each antenna-tag pair,the reader 5)Flick 6)Swipe Left 7)Swipe Right 8)Punch (SL) (SR) obtains a sequence of RF phase values called a phase profile P門 from the tag's responses over time.To tradeoff between accuracy Fig.1.Example finger micromovements. and robustness in terms of matching resolution,we propose a two phase approach,including coarse-grained filtering and fine- grained matching.To tackle the variation of template phase in a more natural approach,such that the user can directly profiles at different positions,we propose a phase-model-based manipulate the virtual or real objects in the air.This paper con- solution to reconstruct the template phase profiles based on cerns multi-touch in the air,i.e.,the recognition of concurrent the exact locations.Experiment results show that we achieve micromovements using Radio Frequency(RF)signals in RFID an average accuracy of 92.1%under various moving speeds, systems [4]-9.In particular,we consider the concurrent orientation deviations,and so on. micromovements of multiple fingers such as zoom in/out, Index Terms-Passive RFID,RF Signal,micromovement rotate left/right,flick,swipe left/right and punch,as illustrated recognition,prototype design. in Fig.1.This is useful for many applications that requires human-computer interaction through fine-grained concurrent I.INTRODUCTION finger micromovements,such as motion sensing games.For example,a user can manipulate a virtual object with his finger A.Motivation micromovements,such as rotating or stretching the object. TOWADAYS,the human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards and touchpads to the brand-new B.Summary and Limitations of Prior Art approaches of performing interactions in the air.The users can Existing motion sensing technologies use either cameras perform the interactions with the computer using their arms, or sensors.Microsoft Kinect [1]and Leap Motion [3]con- legs and even fingers [1]-[3].In this way,the applications trollers use cameras to capture human motions based on vision of virtual reality and augmented reality can be supported technologies.The key limitation of camera based schemes is that they are more or less affected by the viewing angle Manuscript received December 13.2016:revised July 18.2017:accepted and light condition.Nintendo Wii [2]video game systems November 5,2017;approved by IEEE/ACM TRANSACTIONS ON NETWORK- ING Editor X.Zhou.This work was supported in part by the National Natural use wearable sensors based on infrared technologies.The key Science Foundation of China under Grant 61472185,Grant 61472184,Grant limitation of sensor based schemes is that the sensors are 61373129.Grant 61321491,and Grant 61502224.in part by the Jiangsu Nat- ural Science Foundation under Grant BK20151390,in part by the Fundamental often too big to be conveniently wear.Recently RF-IDraw Research Funds for the Central Universities under Grant 020214380035,in is proposed to use a 2-dimensional array of RFID antennas to part by the National Science Foundation under Grant CNS-1421407,in part by track the movement trajectory of a finger,which is attached the Jiangsu Innovation and Entrepreneurship (Shuangchuang)Program,and with an RFID tag [10].It constructs an efficient beam for in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization.(Corresponding authors:Alex X.Liu:Sanglu Lu.) detecting the finger moving direction by intersecting the beams The authors are with the State Key Laboratory for Novel Software of multiple antennas.However,RF-IDraw is designed to track Technology,Nanjing University.Nanjing 210023,China (e-mail: a fairly large range movement of one finger,e.g.,in the size of Ixie@nju.edu.cn: wangcyu217@dislab.nju.edu.cn; alexliu@nju.edu.cn: sunjq@dislab.nju.edu.cn:sanglu@nju.edu.cn). 20~30cm.It does not work well for tracking the concurrent Digital Object Identifier 10.1109/TNET.2017.2772781 movements of multiple fingers because its median accuracy 1063-66922017 IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE/ACM TRANSACTIONS ON NETWORKING 1 Multi-Touch in the Air: Concurrent Micromovement Recognition Using RF Signals Lei Xie , Member, IEEE, Chuyu Wang, Student Member, IEEE, Alex X. Liu, Senior Member, IEEE, Jianqiang Sun, and Sanglu Lu, Member, IEEE Abstract— The human–computer interactions have moved from the conventional approaches of entering inputs into the keyboards/touchpads to the brand-new approaches of performing interactions in the air. In this paper, we propose RF-glove, a system that recognizes concurrent multiple finger micromovement using RF signals, so as to realize the vision of "multi-touch in the air." It uses a commercial-off-the-shelf (COTS) RFID reader with three antennas and five COTS tags attached to the five fingers of a glove, one tag per finger. During the process of a user performing finger micromovements, we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag. For each antenna–tag pair, the reader obtains a sequence of RF phase values called a phase profile from the tag’s responses over time. To tradeoff between accuracy and robustness in terms of matching resolution, we propose a two phase approach, including coarse-grained filtering and finegrained matching. To tackle the variation of template phase profiles at different positions, we propose a phase-model-based solution to reconstruct the template phase profiles based on the exact locations. Experiment results show that we achieve an average accuracy of 92.1% under various moving speeds, orientation deviations, and so on. Index Terms— Passive RFID, RF Signal, micromovement recognition, prototype design. I. INTRODUCTION A. Motivation NOWADAYS, the human-computer interactions have moved from the conventional approaches of entering inputs into the keyboards and touchpads to the brand-new approaches of performing interactions in the air. The users can perform the interactions with the computer using their arms, legs and even fingers [1]–[3]. In this way, the applications of virtual reality and augmented reality can be supported Manuscript received December 13, 2016; revised July 18, 2017; accepted November 5, 2017; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor X. Zhou. This work was supported in part by the National Natural Science Foundation of China under Grant 61472185, Grant 61472184, Grant 61373129, Grant 61321491, and Grant 61502224, in part by the Jiangsu Natural Science Foundation under Grant BK20151390, in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035, in part by the National Science Foundation under Grant CNS-1421407, in part by the Jiangsu Innovation and Entrepreneurship (Shuangchuang) Program, and in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization. (Corresponding authors: Alex X. Liu; Sanglu Lu.) The authors are with the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China (e-mail: lxie@nju.edu.cn; wangcyu217@dislab.nju.edu.cn; alexliu@nju.edu.cn; sunjq@dislab.nju.edu.cn; sanglu@nju.edu.cn). Digital Object Identifier 10.1109/TNET.2017.2772781 Fig. 1. Example finger micromovements. in a more natural approach, such that the user can directly manipulate the virtual or real objects in the air. This paper concerns multi-touch in the air, i.e., the recognition of concurrent micromovements using Radio Frequency (RF) signals in RFID systems [4]–[9]. In particular, we consider the concurrent micromovements of multiple fingers such as zoom in/out, rotate left/right, flick, swipe left/right and punch, as illustrated in Fig. 1. This is useful for many applications that requires human-computer interaction through fine-grained concurrent finger micromovements, such as motion sensing games. For example, a user can manipulate a virtual object with his finger micromovements, such as rotating or stretching the object. B. Summary and Limitations of Prior Art Existing motion sensing technologies use either cameras or sensors. Microsoft Kinect [1] and Leap Motion [3] controllers use cameras to capture human motions based on vision technologies. The key limitation of camera based schemes is that they are more or less affected by the viewing angle and light condition. Nintendo Wii [2] video game systems use wearable sensors based on infrared technologies. The key limitation of sensor based schemes is that the sensors are often too big to be conveniently wear. Recently RF-IDraw is proposed to use a 2-dimensional array of RFID antennas to track the movement trajectory of a finger, which is attached with an RFID tag [10]. It constructs an efficient beam for detecting the finger moving direction by intersecting the beams of multiple antennas. However, RF-IDraw is designed to track a fairly large range movement of one finger, e.g., in the size of 20∼30cm. It does not work well for tracking the concurrent movements of multiple fingers because its median accuracy 1063-6692 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING is 3.7cm,which means that the accuracy of tracking two REID fingers could be 7.4cm,but finger movements are typically 2cm to 5cm.Furthermore,the deployment cost of RF-IDraw RFID is relatively expensive as it requires an antenna array of eight Antenna antennas and two RFID readers.Similarly,RFID localization eration schemes do not work well for recognizing concurrent multi- ane finger micromovements because the location accuracy is not Hands with multiple enough.For example,the state-of-the-art localization schemes RFID tags PinIt achieves an accuracy of 16cm at 90 percentile [11]. and Tagoram achieves an accuracy with a median error dis- tance of 6.35cm [12].In summary,the above RFID-based localization schemes,including RF-IDraw,are not suitable for micromovement recognition,as they mainly focus on the absolute tag positioning rather than the relative movement pattern of multiple tags.As a matter of fact,to achieve more accurate performance in the micromovement recognition,we should focus on the phase variation pattern caused by the micromovement of multiple fingers,instead of capturing the location variation of multiple fingers,since the former metric captures the micromovement in much more fine granularity than the latter. (b) Fig.2.System Overview.(a)Antenna deployment.(b)Tag deployment. C.Proposed Approach In this paper,we propose RF-Glove,a concurrent multi- finger micromovement recognition system based on RF sig- In other words,each different type of multi-finger micromove- nals.RF-Glove uses a commercial off-the-shelf(COTS)RFID ments can be characterized by different RF phase variations reader with 3 antennas and five EPCglobal C1G2 standard Thus,by capturing the distinguishing RF phase variation passive tags attached to the five fingers of a glove,one patterns,we can recognize different multi-finger micromove- tag per finger.The three antennas form two antenna pairs, ments.Our RF-micromovement model fundamentally explains which are placed in a mutually orthogonal manner on a flat why multi-finger micromovements can be recognized based on plane.Fig.2 shows the overview of our system with the 3 phase variations from RF signals. antennas deployed on the office ceiling.In performing multi- finger micromovements,we let the RFID reader continuously D.Technical Challenges and Solutions interrogate these tags and obtain the backscattered RF signals There are several technical challenges we need to address in from each tag.For each antenna-tag pair,the reader obtains a this paper.The first challenge is to properly tradeoff between sequence of RF phase values called a phase profile.For each accuracy and robustness in terms of matching resolution. type of multi-finger micromovement,we obtain a set of 3x 5 Given a testing set of phase profiles and a few template phase profiles.Given the phase profile set of a testing multi- sets of phase profiles for the micromovement,we need to finger micromovement,we compare the corresponding set of find the template set that the testing set matches the best. phase profiles with the templates of each type of multi-finger If the matching resolution is too high,then the matching micromovement to find the most similar template. robustness is too low due to the inherent unstableness in multi- To understand how RF signals vary with multi-finger micro- finger micromovements.If the matching resolution is too low, movements,in this paper,we propose a 3D positioning model then the matching accuracy is too low due to the inherent and a RF-micromovement model,respectively,to depict the common characteristics among different types of multi-finger relationship between the multi-finger movement and the RF- micromovements.To address this challenge,we propose a signals.Specifically,to recognize the large range movement, two-phase approach to this matching problem.In this first such as the swipe and punch,and locate the position of phase,we perform a coarse-grained filtering to identify some the hand during the small-range micromovement,such as the template sets that the testing set should not be matched to. zoom in/out and flick,we propose a 3D positioning model by referring to the moving status of the fingers and the to continuously locate the tags'positions for further micro- variation trend of the phase profile.In the second phase, movement recognition.To depict the relationship between we perform a fine-grained matching to match the testing set the phase variation and the multi-finger micromovement, to one of the remaining template sets,by referring to the we propose a RF-micromovement model that quantifies the details of phase profiles with time warping.Thus,we can use relationship between RF signals and micromovements.Our different matching resolutions to tradeoff between accuracy RF-micromovement model shows that RF phase variations and robustness. and multi-finger micromovement present a linear relationship, The second challenge is to tackle the variation of tem- when it is performed in the central beam of the antenna.plate phase profiles at different positions.It is observed that
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 2 IEEE/ACM TRANSACTIONS ON NETWORKING is 3.7cm, which means that the accuracy of tracking two fingers could be 7.4cm, but finger movements are typically 2cm to 5cm. Furthermore, the deployment cost of RF-IDraw is relatively expensive as it requires an antenna array of eight antennas and two RFID readers. Similarly, RFID localization schemes do not work well for recognizing concurrent multi- finger micromovements because the location accuracy is not enough. For example, the state-of-the-art localization schemes PinIt achieves an accuracy of 16cm at 90 percentile [11], and Tagoram achieves an accuracy with a median error distance of 6.35cm [12]. In summary, the above RFID-based localization schemes, including RF-IDraw, are not suitable for micromovement recognition, as they mainly focus on the absolute tag positioning rather than the relative movement pattern of multiple tags. As a matter of fact, to achieve more accurate performance in the micromovement recognition, we should focus on the phase variation pattern caused by the micromovement of multiple fingers, instead of capturing the location variation of multiple fingers, since the former metric captures the micromovement in much more fine granularity than the latter. C. Proposed Approach In this paper, we propose RF-Glove, a concurrent multi- finger micromovement recognition system based on RF signals. RF-Glove uses a commercial off-the-shelf (COTS) RFID reader with 3 antennas and five EPCglobal C1G2 standard passive tags attached to the five fingers of a glove, one tag per finger. The three antennas form two antenna pairs, which are placed in a mutually orthogonal manner on a flat plane. Fig. 2 shows the overview of our system with the 3 antennas deployed on the office ceiling. In performing multi- finger micromovements, we let the RFID reader continuously interrogate these tags and obtain the backscattered RF signals from each tag. For each antenna-tag pair, the reader obtains a sequence of RF phase values called a phase profile. For each type of multi-finger micromovement, we obtain a set of 3 × 5 phase profiles. Given the phase profile set of a testing multi- finger micromovement, we compare the corresponding set of phase profiles with the templates of each type of multi-finger micromovement to find the most similar template. To understand how RF signals vary with multi-finger micromovements, in this paper, we propose a 3D positioning model and a RF-micromovement model, respectively, to depict the relationship between the multi-finger movement and the RFsignals. Specifically, to recognize the large range movement, such as the swipe and punch, and locate the position of the hand during the small-range micromovement, such as the zoom in/out and flick, we propose a 3D positioning model to continuously locate the tags’ positions for further micromovement recognition. To depict the relationship between the phase variation and the multi-finger micromovement, we propose a RF-micromovement model that quantifies the relationship between RF signals and micromovements. Our RF-micromovement model shows that RF phase variations and multi-finger micromovement present a linear relationship, when it is performed in the central beam of the antenna. Fig. 2. System Overview. (a) Antenna deployment. (b) Tag deployment. In other words, each different type of multi-finger micromovements can be characterized by different RF phase variations. Thus, by capturing the distinguishing RF phase variation patterns, we can recognize different multi-finger micromovements. Our RF-micromovement model fundamentally explains why multi-finger micromovements can be recognized based on phase variations from RF signals. D. Technical Challenges and Solutions There are several technical challenges we need to address in this paper. The first challenge is to properly tradeoff between accuracy and robustness in terms of matching resolution. Given a testing set of phase profiles and a few template sets of phase profiles for the micromovement, we need to find the template set that the testing set matches the best. If the matching resolution is too high, then the matching robustness is too low due to the inherent unstableness in multi- finger micromovements. If the matching resolution is too low, then the matching accuracy is too low due to the inherent common characteristics among different types of multi-finger micromovements. To address this challenge, we propose a two-phase approach to this matching problem. In this first phase, we perform a coarse-grained filtering to identify some template sets that the testing set should not be matched to, by referring to the moving status of the fingers and the variation trend of the phase profile. In the second phase, we perform a fine-grained matching to match the testing set to one of the remaining template sets, by referring to the details of phase profiles with time warping. Thus, we can use different matching resolutions to tradeoff between accuracy and robustness. The second challenge is to tackle the variation of template phase profiles at different positions. It is observed that
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS when the human subject performs the micromovement at the are attached to a controlling ball to detect the motions of ball positions out of the central beams of the antennas,the phase rotation from users.Compared with our RF-Glove system, profiles for the same micromovement might be different to a the tags in Tagball follow the same movement trace where certain extent at different positions.Hence,it is inaccurate to the tags in RF-Glove may follow different movement traces. directly match the testing set of phase profiles to the original RF-IDraw [10]uses a 2-dimensional array of RFID antennas template set of phase profiles.To address this challenge,we to track the movement trajectory of one finger attached with propose a solution to reconstruct the template phase profiles an RFID tag,so that it can reconstruct the trajectory shape based on the exact locations.We first propose a 3D positioning of the specified finger.However,RF-IDraw is designed to method based on the AoA method to figure out the locations track a fairly large range movement of one finger,e.g.,in of multiple fingers.Based on the fingers'location,we propose the size of 20~30cm.It does not work well for tracking the a model to depict the relationship between the phase variation concurrent movements of multiple fingers because its median and the specified movement.We further derive the correspond- accuracy is 3.7cm,which means that the accuracy of tracking ing template phase profiles based on the exact locations. two fingers could be 7.4cm,but finger movements are typically We make four key contributions in this paper.First,we 2cm to 5cm.Furthermore,the deployment cost of RF-IDraw propose RF-Glove,an RF signal based concurrent micromove- is relatively expensive as it requires an antenna array of ment recognition system.Second,we propose a 3D position- eight antennas and two RFID readers.Different from the ing model and a RF-micromovement model,respectively,to positioning-based techniques from RF-IDraw,in this paper,to depict the relationship between the multi-finger movement and achieve more accurate performance in micromovement recog- the RF-signals.Third,we propose a phase profiling based nition,we directly investigate the phase variation pattern from approach to RF signal based multi-finger micromovement the concurrent micromovement of multiple fingers,instead of recognition.Last,we implemented RF-Glove using COTS capturing the location variation of multiple fingers,since the RFID systems and evaluated its performance in realistic set- former metric captures the micromovement in much more fine tings.Experiment results show that we achieve an average granularity than the latter. accuracy of 92.1%under various moving speeds,orientation deviations.etc. III.MODELING RF SIGNAL VARIATIONS AND MULTI-FINGER MICROMOVEMENTS II.RELATED WORK Like the functionalities of the general purpose touch pad, the scheme of"multi-touch in the air"should also have both RFID-Based Localization:Prior work on RFID-based local- the positioning and gesture-recognition functionalities.The ization primarily rely on RSSI (Received Signal Strength) positioning functionality aims to locate the tagged fingers information [13],[14]to acquire the absolute location of an in a coarse-grained manner.In this way,we are able to object.State-of-the-art systems use phase value to estimate the easily recognize the large-range movement of the tagged absolute location of an object with higher accuracy [11],[12], fingers caused by the arm movement.The gesture-recognition [15]-[21].By deploying multiple antennas and measuring the functionality aims to further recognize the micromovement of phase difference between the received signals at different multiple tagged fingers in a fine-grained manner. antennas,these systems can effectively reduce the localization Therefore,to understand how RF-signals vary with large- error to a few centimeters.Further,PinIt exploits multi-path range movement,we propose a 3D positioning model that effect to accurately locate RFIDs by using synthetic aperture quantifies the relationship between the RF signal and the radar created via antenna motion to extract multi-path profiles position of tagged fingers in the 3-dimensional space.To for accurate localization [11].Tagoram exploits tag mobility to understand how RF signals vary with multi-finger micro- build a virtual antenna array,and uses differential augmented movements,we propose an RF micromovement model that hologram to facilitate the instant tracking of a mobile RFID quantifies the relationship between RF signals and multi-finger tag [12].While the above work mainly focuses on absolute micromovements.It shows that each different type of multi- object localization.Spatial-Temporal Phase Profiling (STPP) finger micromovements can be characterized by different RF is proposed for the relative localization of RFID tags [22]. phase variation patterns.Thus,by capturing the distinguishing Liu et al.[20]propose a pose sensing system called Tag- RF phase variation patterns,we can recognize different multi- Compass that uses a single tag to determine the orientation finger micromovements. as well as the position of the associated object.A completely We use the commercial RFID reader ImpinJ R420 and Laird different method based on the polarization properties of the S9028 antenna to receive RF signals.Laird S9028 antenna pro- RF waves is exploited to achieve fine-grained pose sensing. vides a consistent and continuous reading zone with circular RFID-Based Motion Tracking:Prior activity sensing sys- polarization.As shown in Figure 2(a),we deploy three anten- tems propose various approaches to recognize gestures for nas on the room ceiling,say A,B and C.The antenna pair activity sensing.These systems can be primarily classified into AB and AC are deployed in a mutually orthogonal fashion vision-based,infrared-based,electric field-based and wearable along the X-axis and Y-axis,respectively.By leveraging the approaches [23]-[25].RFID systems have recently been used Angle of Arrival (AoA)positioning method,the AoA from for trajectory tracking [10],[26].[27]and motion tracking the antenna pair AB can differentiate the movement along the [28]-[32].Lin et al.[29]proposed a 3D human-computer X-axis,while the AoA from the antenna pair AC can differ- interaction system called Tagball,where multiple passive tags entiate the movement along the Y-axis.We use Alien 9640
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 3 when the human subject performs the micromovement at the positions out of the central beams of the antennas, the phase profiles for the same micromovement might be different to a certain extent at different positions. Hence, it is inaccurate to directly match the testing set of phase profiles to the original template set of phase profiles. To address this challenge, we propose a solution to reconstruct the template phase profiles based on the exact locations. We first propose a 3D positioning method based on the AoA method to figure out the locations of multiple fingers. Based on the fingers’ location, we propose a model to depict the relationship between the phase variation and the specified movement. We further derive the corresponding template phase profiles based on the exact locations. We make four key contributions in this paper. First, we propose RF-Glove, an RF signal based concurrent micromovement recognition system. Second, we propose a 3D positioning model and a RF-micromovement model, respectively, to depict the relationship between the multi-finger movement and the RF-signals. Third, we propose a phase profiling based approach to RF signal based multi-finger micromovement recognition. Last, we implemented RF-Glove using COTS RFID systems and evaluated its performance in realistic settings. Experiment results show that we achieve an average accuracy of 92.1% under various moving speeds, orientation deviations, etc. II. RELATED WORK RFID-Based Localization: Prior work on RFID-based localization primarily rely on RSSI (Received Signal Strength) information [13], [14] to acquire the absolute location of an object. State-of-the-art systems use phase value to estimate the absolute location of an object with higher accuracy [11], [12], [15]–[21]. By deploying multiple antennas and measuring the phase difference between the received signals at different antennas, these systems can effectively reduce the localization error to a few centimeters. Further, PinIt exploits multi-path effect to accurately locate RFIDs by using synthetic aperture radar created via antenna motion to extract multi-path profiles for accurate localization [11]. Tagoram exploits tag mobility to build a virtual antenna array, and uses differential augmented hologram to facilitate the instant tracking of a mobile RFID tag [12]. While the above work mainly focuses on absolute object localization, Spatial-Temporal Phase Profiling (STPP) is proposed for the relative localization of RFID tags [22]. Liu et al. [20] propose a pose sensing system called TagCompass that uses a single tag to determine the orientation as well as the position of the associated object. A completely different method based on the polarization properties of the RF waves is exploited to achieve fine-grained pose sensing. RFID-Based Motion Tracking: Prior activity sensing systems propose various approaches to recognize gestures for activity sensing. These systems can be primarily classified into vision-based, infrared-based, electric field-based and wearable approaches [23]–[25]. RFID systems have recently been used for trajectory tracking [10], [26], [27] and motion tracking [28]–[32]. Lin et al. [29] proposed a 3D human-computer interaction system called Tagball, where multiple passive tags are attached to a controlling ball to detect the motions of ball rotation from users. Compared with our RF-Glove system, the tags in Tagball follow the same movement trace where the tags in RF-Glove may follow different movement traces. RF-IDraw [10] uses a 2-dimensional array of RFID antennas to track the movement trajectory of one finger attached with an RFID tag, so that it can reconstruct the trajectory shape of the specified finger. However, RF-IDraw is designed to track a fairly large range movement of one finger, e.g., in the size of 20∼30cm. It does not work well for tracking the concurrent movements of multiple fingers because its median accuracy is 3.7cm, which means that the accuracy of tracking two fingers could be 7.4cm, but finger movements are typically 2cm to 5cm. Furthermore, the deployment cost of RF-IDraw is relatively expensive as it requires an antenna array of eight antennas and two RFID readers. Different from the positioning-based techniques from RF-IDraw, in this paper, to achieve more accurate performance in micromovement recognition, we directly investigate the phase variation pattern from the concurrent micromovement of multiple fingers, instead of capturing the location variation of multiple fingers, since the former metric captures the micromovement in much more fine granularity than the latter. III. MODELING RF SIGNAL VARIATIONS AND MULTI-FINGER MICROMOVEMENTS Like the functionalities of the general purpose touch pad, the scheme of “multi-touch in the air” should also have both the positioning and gesture-recognition functionalities. The positioning functionality aims to locate the tagged fingers in a coarse-grained manner. In this way, we are able to easily recognize the large-range movement of the tagged fingers caused by the arm movement. The gesture-recognition functionality aims to further recognize the micromovement of multiple tagged fingers in a fine-grained manner. Therefore, to understand how RF-signals vary with largerange movement, we propose a 3D positioning model that quantifies the relationship between the RF signal and the position of tagged fingers in the 3-dimensional space. To understand how RF signals vary with multi-finger micromovements, we propose an RF micromovement model that quantifies the relationship between RF signals and multi-finger micromovements. It shows that each different type of multi- finger micromovements can be characterized by different RF phase variation patterns. Thus, by capturing the distinguishing RF phase variation patterns, we can recognize different multi- finger micromovements. We use the commercial RFID reader ImpinJ R420 and Laird S9028 antenna to receive RF signals. Laird S9028 antenna provides a consistent and continuous reading zone with circular polarization. As shown in Figure 2(a), we deploy three antennas on the room ceiling, say A, B and C. The antenna pair AB and AC are deployed in a mutually orthogonal fashion along the X-axis and Y -axis, respectively. By leveraging the Angle of Arrival (AoA) positioning method, the AoA from the antenna pair AB can differentiate the movement along the X-axis, while the AoA from the antenna pair AC can differentiate the movement along the Y -axis. We use Alien 9640
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination IEEE/ACM TRANSACTIONS ON NETWORKING general-purpose tags,which are EPC C1G2 standards compli- Z◆4Y ant.We attach five RFID tags to the five fingers of a glove,one tag per finger,as shown in Figure 2(b).In performing multi- P(X,0,) finger micromovements,we let the RFID reader continuously 08i0,0,0) interrogate these tags and obtain RF signals from each tag via three antennas. Operation Plane 口0' A.3D Positioning Model p-0,0,h (x,y,-h) When the human subject performs the multi-touch gesture in the air with the tagged fingers,he/she usually performs the Fig.3.The position of the tag P on the operation plane. following two kinds of movement:1)Large-range movement: the human subject performs the movement with fairly large range in the 3-dimensional space,such as swipe left/right and punch.The moving range is usually greater than half of the wave length (i.e.,about 17cm)so that the phase changes of the RFID tags exceed a complete period.2)Small-range micromovement:the human subject performs the movement with very small range in the 3-dimensional space,such as zoom in/out,rotate left/right,and flick.The moving range is less than half of the wave length(i.e.,about 17cm)so that the 10 phase changes of the RFID tags does not exceed a complete X axis period. Fig.4.The hyperbola:the intersection between the conical surface and the Therefore,for the large-range movement,since the phase operation plane. change exceeds a complete period,it is neither accurate nor necessary to recognize the movement by phase changes. Instead,we can leverage the 3D positioning method to effec- as (,y,-h).Thus POll =vx2 +2+h2.Since the angle tively recognize the large-range movement.For the small- ∠POP'=a,then range micromovement,since the position change of the tagged ‖PO=IPOll cosa. (1) fingers is rather small (it is usually less than 5cm).thus we rely on the phase changes to recognize the small-range micromove- Hence,as P'Ol =Eq.(1)is equivalent to ment.Nevertheless,the phase changes of the micromovement l=Vx2+y2+h2.cosa. (2) also depends on the exact position of the tagged fingers.For example,the phase changes of the same type of micromove- Therefore, ment may vary to a certain extent when it is performed at sin2a·x2-cos2a·y2=h2.cos2a. different positions of the 3-dimensional space.In summary, (3) 3D positioning is essential in recognizing both the large-range It implies that the feasible solution of P on the operational and small-range movement. plane is a hyperbola.We further illustrate the above conclu- Suppose we can build a 3D coordinate system according to sion with an example as shown in Fig.4.Since the angle the operation plane,as shown in Fig.3.The antenna pair is POP'=a,and OP'is collinear with the X-axis,so the deployed along the X-axis,while the origin O is set to the possible trace of P in the 3-dimensional space can be denoted center of the antenna pair.The X-axis and Y-axis are mutually as a conical surface originated from the point O.When the orthogonal and parallel to the operation plane,and the Z-axis conical surface intersects with the operation plane,the possible is orthogonal to the operation plane.Assume that a specified trace of P forms a hyperbola on the operation plane. tag is denoted as a point P=(,y,z)on the operation plane, Suppose the human subject is performing the micromove- the projection of the point P on the X-axis is P.As in ment on the same operation plane,more or less.That is conventional operations of multi-touch in the air,the tagged to say,the distance h between the operation plane and the fingers of the human subjects are separated with a fairly large antenna plane keeps almost constant.As we deploy two mutu- distance (e.g.,150cm~200cm)to the three antennas,whereas ally orthogonal antenna pairs along the X-axis and Y-axis, the antennas are separated with a limited distance (e.g.,20cm respectively,then,according to the two antenna pairs,the ~30cm)to each other,thus we can leverage the Angle of feasible solutions of the tag's position can be estimated as two Arrival(AoA)method to figure out the direction of the tag in hyperbolas intersecting on the operation plane.Thus,we can the 3D space.Then,according to the AoA method,we can estimate the position of the tag by computing the intersections estimate the angle between OP and OP/as a.Assume that of the two hyperbolas.Fig.5 shows an example of positioning the projection of the origin O on the operation plane is O', the tag by computing the intersections between two hyperbolas then the distance between the antenna plane and the operation on the operation plane.Here,the antennas are separated with a plane is h=OO'l=-z.Therefore,the coordinate of horizontal/vertical distance of 20cm,and the distance between O'is (0,0,-h),the coordinate of P can be also denoted the operation plane and the antenna plane is set to 100cm
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 4 IEEE/ACM TRANSACTIONS ON NETWORKING general-purpose tags, which are EPC C1G2 standards compliant. We attach five RFID tags to the five fingers of a glove, one tag per finger, as shown in Figure 2(b). In performing multi- finger micromovements, we let the RFID reader continuously interrogate these tags and obtain RF signals from each tag via three antennas. A. 3D Positioning Model When the human subject performs the multi-touch gesture in the air with the tagged fingers, he/she usually performs the following two kinds of movement: 1) Large-range movement: the human subject performs the movement with fairly large range in the 3-dimensional space, such as swipe left/right and punch. The moving range is usually greater than half of the wave length (i.e., about 17cm) so that the phase changes of the RFID tags exceed a complete period. 2) Small-range micromovement: the human subject performs the movement with very small range in the 3-dimensional space, such as zoom in/out, rotate left/right, and flick. The moving range is less than half of the wave length (i.e., about 17cm) so that the phase changes of the RFID tags does not exceed a complete period. Therefore, for the large-range movement, since the phase change exceeds a complete period, it is neither accurate nor necessary to recognize the movement by phase changes. Instead, we can leverage the 3D positioning method to effectively recognize the large-range movement. For the smallrange micromovement, since the position change of the tagged fingers is rather small (it is usually less than 5cm), thus we rely on the phase changes to recognize the small-range micromovement. Nevertheless, the phase changes of the micromovement also depends on the exact position of the tagged fingers. For example, the phase changes of the same type of micromovement may vary to a certain extent when it is performed at different positions of the 3-dimensional space. In summary, 3D positioning is essential in recognizing both the large-range and small-range movement. Suppose we can build a 3D coordinate system according to the operation plane, as shown in Fig.3. The antenna pair is deployed along the X-axis, while the origin O is set to the center of the antenna pair. The X-axis and Y -axis are mutually orthogonal and parallel to the operation plane, and the Z-axis is orthogonal to the operation plane. Assume that a specified tag is denoted as a point P = (x, y, z) on the operation plane, the projection of the point P on the X-axis is P . As in conventional operations of multi-touch in the air, the tagged fingers of the human subjects are separated with a fairly large distance (e.g., 150cm∼200cm) to the three antennas, whereas the antennas are separated with a limited distance (e.g., 20cm ∼30cm) to each other, thus we can leverage the Angle of Arrival (AoA) method to figure out the direction of the tag in the 3D space. Then, according to the AoA method, we can estimate the angle between OP and OP as α. Assume that the projection of the origin O on the operation plane is O , then the distance between the antenna plane and the operation plane is h = OO = −z. Therefore, the coordinate of O is (0, 0, −h), the coordinate of P can be also denoted Fig. 3. The position of the tag P on the operation plane. Fig. 4. The hyperbola: the intersection between the conical surface and the operation plane. as (x, y, −h). Thus P O = x2 + y2 + h2. Since the angle ∠POP = α, then P O = P O · cos α. (1) Hence, as P O = |x|, Eq.(1) is equivalent to |x| = x2 + y2 + h2 · cos α. (2) Therefore, sin2 α · x2 − cos2 α · y2 = h2 · cos2 α. (3) It implies that the feasible solution of P on the operational plane is a hyperbola. We further illustrate the above conclusion with an example as shown in Fig.4. Since the angle ∠POP = α, and OP is collinear with the X-axis, so the possible trace of P in the 3-dimensional space can be denoted as a conical surface originated from the point O. When the conical surface intersects with the operation plane, the possible trace of P forms a hyperbola on the operation plane. Suppose the human subject is performing the micromovement on the same operation plane, more or less. That is to say, the distance h between the operation plane and the antenna plane keeps almost constant. As we deploy two mutually orthogonal antenna pairs along the X-axis and Y -axis, respectively, then, according to the two antenna pairs, the feasible solutions of the tag’s position can be estimated as two hyperbolas intersecting on the operation plane. Thus, we can estimate the position of the tag by computing the intersections of the two hyperbolas. Fig.5 shows an example of positioning the tag by computing the intersections between two hyperbolas on the operation plane. Here, the antennas are separated with a horizontal/vertical distance of 20cm, and the distance between the operation plane and the antenna plane is set to 100cm.
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination. XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 60 Yaxis Contour lines of 一Hyperbola 1 `、phase values 一Hyperbola2 Target position Antenna position 、、 X axis (xsys,zs)\ ds Ny(xeye,Ze） de 0 1 /Antenna 例 △d Z axis 20 Fig.7.The relationship between△dand△z. Fig.5.Tag positioning:the intersection between two hyperbolas on the variation of phase value A from adjacent phase values as operation plane. follows,where term u=0r+0R+TAc is canceled out: △0= ·×2△d)mod2π (5) Here,Ad is the variation of distance between the tag and the antenna.If the movement Ad is smaller than half a wavelength ≥，ie,about 16.4cm,then, Operation Plane △8= 2×△d ×2π」 (6) X V 入 Thus,we can derive that the movement of the tag towards the antenna is△d=六△， Fig.6.Estimate the parameter h by moving the hand linearly with distance d. For our system deployment,we deploy three antennas,say A,B and C.on the office ceiling,as shown in Fig.2(a).To To figure out the hyperbolas of the specified tag,it is depict the micromovement of the tagged fingers,it is essential essential to estimate the distance h.To estimate the para- to build a 3-dimensional coordinate system.Without loss of meter h,we can let the human subject perform a specified generality,we build a coordinate system by setting the center movement,e.g.,performing the push/pull movement with a of antenna A as the origin,as shown in Fig.7.Then,for specified distance d from time t to t'.Then,according to the antenna A,let Ad be the change of the distance between the distance d,we can enumerate all feasible values of h,and tag and the antenna,.and let△x,△y,and△e be the change of compute the intersection point P(t)and P(t')between the positions for the tag along the three dimensions X.Y,and Z, hyperbolas,respectively,at the start time and end time of the respectively.Suppose a tag moves from position (s,4s,2)to movement.We finally determine the estimate of h when the (re:ye,ze),then△d=V√径+y+z径-√g+y+z径，and corresponding moving distance is most close to d.Fig.6 shows Az=ze-2s.In our system,we require the tagged fingers to an example of estimating the parameter h by moving the hand be separated with the antennas with a fairly large distance of linearly with a specified distance d. more than 150cm.Suppose we let the glove attached with five tags operate within the central beam area of the antenna.Then, B.RF-Micromovement Model we rely on the following theorem to depict the relationship between△dand△z. The phase value of an RF signal describes the degree that Theorem 1:Considering a 3-dimensional space with the received signal offsets from the sent signal,ranging from antenna A as the origin,suppose a tag moves from position 0 to 360 degrees.Let d be the distance between the RFID (s,ys,2s)to (e,ye,ze),then,considering antenna A,we antenna and the tag,the signal traverses a round-trip with a define the change of the distance between the tag and the distance of 2d in each backscatter communication.Thus,the antenna△d=vc+y+2径-√rg+y好+2径，and the phase value 0 output by an RFID reader can be expressed as: change of positions for the tag along the dimension X,Y and ×2d+）mod2π (4) Zare△x=re-ra,△y=ye-ys,and△z=ze-zs: respectively.If each tag moves within its central beam,i.e., where A is the wave length.Besides the RF phase rotation 2s s,2s>ys.ze e,and ze ye,then.Ad is over distance,the reader's transmitter,the tag's reflection approximately equal to Az. characteristic,and the reader's receiver will also introduce Proof:According to the definition of△dand△z, some additional phase rotation,denoted as er,OR and rAc respectively.We use u=0T+0R+rAc to denote this △d 哈+呢+2是-+班+图 (7) diversity term in Equation(1). △z 2e-2s As we focus on the variation of tag positions rather than x2+呢+是-x?++2 (8) absolute positions,for each antenna-tag pair,we compute the (2e-)·(√哈+经+径+√号+好+)
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 5 Fig. 5. Tag positioning: the intersection between two hyperbolas on the operation plane. Fig. 6. Estimate the parameter h by moving the hand linearly with distance d. To figure out the hyperbolas of the specified tag, it is essential to estimate the distance h. To estimate the parameter h, we can let the human subject perform a specified movement, e.g., performing the push/pull movement with a specified distance d from time t to t . Then, according to the distance d, we can enumerate all feasible values of h, and compute the intersection point P(t) and P(t ) between the hyperbolas, respectively, at the start time and end time of the movement. We finally determine the estimate of h when the corresponding moving distance is most close to d. Fig.6 shows an example of estimating the parameter h by moving the hand linearly with a specified distance d. B. RF-Micromovement Model The phase value of an RF signal describes the degree that the received signal offsets from the sent signal, ranging from 0 to 360 degrees. Let d be the distance between the RFID antenna and the tag, the signal traverses a round-trip with a distance of 2d in each backscatter communication. Thus, the phase value θ output by an RFID reader can be expressed as: θ = (2π λ × 2d + μ) mod 2π, (4) where λ is the wave length. Besides the RF phase rotation over distance, the reader’s transmitter, the tag’s reflection characteristic, and the reader’s receiver will also introduce some additional phase rotation, denoted as θT , θR and θT AG respectively. We use μ = θT + θR + θT AG to denote this diversity term in Equation (1). As we focus on the variation of tag positions rather than absolute positions, for each antenna-tag pair, we compute the Fig. 7. The relationship between Δd and Δz. variation of phase value Δθ from adjacent phase values as follows, where term μ = θT + θR + θT AG is canceled out: Δθ = (2π λ × 2Δd) mod 2π. (5) Here, Δd is the variation of distance between the tag and the antenna. If the movement Δd is smaller than half a wavelength λ 2 , i.e., about 16.4cm, then, Δθ = 2 × Δd λ × 2π. (6) Thus, we can derive that the movement of the tag towards the antenna is Δd = λ 4πΔθ. For our system deployment, we deploy three antennas, say A, B and C, on the office ceiling, as shown in Fig.2(a). To depict the micromovement of the tagged fingers, it is essential to build a 3-dimensional coordinate system. Without loss of generality, we build a coordinate system by setting the center of antenna A as the origin, as shown in Fig.7. Then, for antenna A, let Δd be the change of the distance between the tag and the antenna, and let Δx, Δy, and Δz be the change of positions for the tag along the three dimensions X, Y , and Z, respectively. Suppose a tag moves from position (xs, ys, zs) to (xe, ye, ze), then Δd = x2 e + y2 e + z2 e− x2 s + y2 s + z2 s , and Δz = ze − zs. In our system, we require the tagged fingers to be separated with the antennas with a fairly large distance of more than 150cm. Suppose we let the glove attached with five tags operate within the central beam area of the antenna. Then, we rely on the following theorem to depict the relationship between Δd and Δz. Theorem 1: Considering a 3-dimensional space with antenna A as the origin, suppose a tag moves from position (xs, ys, zs) to (xe, ye, ze), then, considering antenna A, we define the change of the distance between the tag and the antenna Δd = x2 e + y2 e + z2 e − x2 s + y2 s + z2 s , and the change of positions for the tag along the dimension X, Y and Z are Δx = xe − xs, Δy = ye − ys, and Δz = ze − zs, respectively. If each tag moves within its central beam, i.e., zs xs, zs ys, ze xe, and ze ye, then, Δd is approximately equal to Δz. Proof: According to the definition of Δd and Δz, Δd Δz = x2 e + y2 e + z2 e − x2 s + y2 s + z2 s ze − zs (7) = x2 e + y2 e + z2 e − x2 s + y2 s + z2 s (ze − zs) · ( x2 e + y2 e + z2 e + x2 s + y2 s + z2 s ) (8)
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination. IEEE/ACM TRANSACTIONS ON NETWORKING Considering antenna A,as each tag moves within its central beam,we have 2s≥xg,zs>ys,ze≥xe,and ze≥e 3D Positioning Therefore, Small Rang Large Range Moving Movement Position-based. Recognition A4≈好-)+（妮-）+(2-边 Range Recognition Results (9 △z (2e-za)·(2e+2s) Unexpected Body Template Phase =1+ 、E一x后一2 Movements Elimination Profile Construction (10) Testing Template 2径-2 2-2 Phase Profile Template Phase Moreover, Extraction Profile Reconstructior 哈- ,=e十工.e-= Te十Ts△T 2-Phase phas (11) 诏一 Matchin 2e十2s2e-2s Ze 2a 2 Asze+za>re+zg,thus#→0，while￥is a constant, Fig.8.System overview 2e十2。 therefore,-号 Ie-zi →0.Similarly,gg 22-2图 →O.Therefore 是≈1.Thus,approximately,.△d≈△2z. ◇ along a certain time window,thus the matrix V[5,l]is already Note that the three antennas A,B and C are deployed sufficient enough to depict the concurrent micromovement. close to each other.and the tagged fingers are required to be separated with the antennas with a fairly large distance of more IV.SYSTEM DESIGN than 150cm,the corresponding results should be very similar. The system is mainly composed of several components,as Therefore,although△d,△x,△y,and△e are defined in their shown in Fig.8.After receiving the RF signals from multiple respective local coordinate systems,as they describe relative tagged fingers,we first leverage 3D positioning to verify distances,these values remain the same,the conclusion is also whether the movement belongs to the small-range micromove- applicable to antenna B and C.Fig.7 illustrates the calculation ment or large-range movement.If it is large-range movement, of Ad and Az,from which we can intuitively observe that we then use the position-based recognition method to perform △d≈△z.Recall that△d=△A.As△d≈△z,we derive movement recognition.Otherwise,we then leverage the phase that the RF phase variations,i.e.,A,and the multi-finger profiling-based method to perform microvement recognition. micromovement in the Z-dimension,i.e.,Az,have a linear Specifically,we use Phase Profile Extraction to extract the relationship as follows:△z≈六△9. testing phase profiles.While dealing with the unexpected When we zoom into the micromovement of one finger, movement introduced into the micromovement,such as the it consists of a number of tiny movements where each of arm movement,we use Unexpected Body Movement Elimina- them may have different directions.We decompose the micro- tion to tackle this issue.Moreover,we use Template Phase movement of each finger into I number of tiny movements Profile Construction to construct the template phase profiles. along the time dimension equally.We represent each tiny While dealing with the phase profile variation at different movement by a vector.As we have five fingers,we obtain a locations,we use Template Phase Profile Reconstruction to matrix reconstruct the template phase profiles based on the exact location.After obtaining the testing phase profiles,we use V[5, (12) 2-Phase Phase Profile Matching to find the template phase profiles that the testing phase profiles match the best.Without V5 loss of generality,in the following sections,we mainly focus Here each vector Vi,is described by(△x,△y,△z). on the recognition of micromovements,such as zoom in/out, As our above model shows that RF phase variations and rotate left/right,and flick,since the large-range movements multi-finger micromovement in the 2-dimension have a linear can be recognized via positioning-based method. relationship,we can characterize a multi-finger micromove- ment directly using phase profile changes.As each Vi.j in A.Template Phase Profile Construction Eq.(l2)denotes(△xr,△y,△z,we can use the phase change A0ii to denote the micromovement in the Z-dimension,i.e., We first present our method of constructing the template Az.Thus,we can directly use a similar matrix V[5,l]to phase profile of each multi-finger micromovement type. characterize a multi-finger micromovement. Phase Curve Stitching:As phase is a periodic function that repeats every wavelength in the distance of signal propagation, △01.1 △01.1 the phase may instantaneously drops from 360 to 0 during the V'[5,I= 13) process of performing a micromovement.This leads to the △05.1 loss of continuity in capturing the continuous micromovement. To address this issue,we stitch the discontinued phase curves Although we can only capture the micromovement in the Z together by removing the 360 degree differences between these dimension,as the multiple fingers have concurrent micro- instantaneously dropping points. movement in the 3-dimensional space,we can capture the Phase Profile Construction:We obtain the phase values of 2-dimensional micromovement of five fingers simultaneously all tags and extract the phase profiles during the process of
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 6 IEEE/ACM TRANSACTIONS ON NETWORKING Considering antenna A, as each tag moves within its central beam, we have zs xs, zs ys, ze xe, and ze ye. Therefore, Δd Δz ≈ (x2 e − x2 s)+(y2 e − y2 s )+(z2 e − z2 s ) (ze − zs) · (ze + zs) (9) = 1+ x2 e − x2 s z2 e − z2 s + y2 e − y2 s z2 e − z2 s (10) Moreover, x2 e − x2 s z2 e − z2 s = xe + xs ze + zs · xe − xs ze − zs = xe + xs ze + zs · Δx Δz . (11) As ze+zs xe+xs, thus xe+xs ze+zs → 0, while Δx Δz is a constant, therefore, x2 e−x2 s z2 e−z2 s → 0. Similarly, y2 e−y2 s z2 e−z2 s → 0. Therefore, Δd Δz ≈ 1. Thus, approximately, Δd ≈ Δz. Note that the three antennas A, B and C are deployed close to each other, and the tagged fingers are required to be separated with the antennas with a fairly large distance of more than 150cm, the corresponding results should be very similar. Therefore, although Δd, Δx, Δy, and Δz are defined in their respective local coordinate systems, as they describe relative distances, these values remain the same, the conclusion is also applicable to antenna B and C. Fig. 7 illustrates the calculation of Δd and Δz, from which we can intuitively observe that Δd ≈ Δz. Recall that Δd = λ 4π Δθ. As Δd ≈ Δz, we derive that the RF phase variations, i.e., Δθ, and the multi-finger micromovement in the Z-dimension, i.e., Δz, have a linear relationship as follows: Δz ≈ λ 4πΔθ. When we zoom into the micromovement of one finger, it consists of a number of tiny movements where each of them may have different directions. We decompose the micromovement of each finger into l number of tiny movements along the time dimension equally. We represent each tiny movement by a vector. As we have five fingers, we obtain a matrix V [5, l] = ⎛ ⎜⎝ V1,1 ··· V1,l . . . . . . . . . V5,1 ··· V5,l ⎞ ⎟⎠. (12) Here each vector Vi,j is described by (Δx, Δy, Δz). As our above model shows that RF phase variations and multi-finger micromovement in the Z-dimension have a linear relationship, we can characterize a multi-finger micromovement directly using phase profile changes. As each Vi,j in Eq. (12) denotes (Δx, Δy, Δz), we can use the phase change Δθi,j to denote the micromovement in the Z-dimension, i.e., Δz. Thus, we can directly use a similar matrix V [5, l] to characterize a multi-finger micromovement. V [5, l] = ⎛ ⎜⎝ Δθ1,1 ··· Δθ1,l . . . . . . . . . Δθ5,1 ··· Δθ5,l ⎞ ⎟⎠. (13) Although we can only capture the micromovement in the Z dimension, as the multiple fingers have concurrent micromovement in the 3-dimensional space, we can capture the Z- dimensional micromovement of five fingers simultaneously Fig. 8. System overview. along a certain time window, thus the matrix V [5, l] is already sufficient enough to depict the concurrent micromovement. IV. SYSTEM DESIGN The system is mainly composed of several components, as shown in Fig.8. After receiving the RF signals from multiple tagged fingers, we first leverage 3D positioning to verify whether the movement belongs to the small-range micromovement or large-range movement. If it is large-range movement, we then use the position-based recognition method to perform movement recognition. Otherwise, we then leverage the phase profiling-based method to perform microvement recognition. Specifically, we use Phase Profile Extraction to extract the testing phase profiles. While dealing with the unexpected movement introduced into the micromovement, such as the arm movement, we use Unexpected Body Movement Elimination to tackle this issue. Moreover, we use Template Phase Profile Construction to construct the template phase profiles. While dealing with the phase profile variation at different locations, we use Template Phase Profile Reconstruction to reconstruct the template phase profiles based on the exact location. After obtaining the testing phase profiles, we use 2-Phase Phase Profile Matching to find the template phase profiles that the testing phase profiles match the best. Without loss of generality, in the following sections, we mainly focus on the recognition of micromovements, such as zoom in/out, rotate left/right, and flick, since the large-range movements can be recognized via positioning-based method. A. Template Phase Profile Construction We first present our method of constructing the template phase profile of each multi-finger micromovement type. Phase Curve Stitching: As phase is a periodic function that repeats every wavelength in the distance of signal propagation, the phase may instantaneously drops from 360 to 0 during the process of performing a micromovement. This leads to the loss of continuity in capturing the continuous micromovement. To address this issue, we stitch the discontinued phase curves together by removing the 360 degree differences between these instantaneously dropping points. Phase Profile Construction: We obtain the phase values of all tags and extract the phase profiles during the process of
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS each micromovement as follows.We take multiple samples B.Phase Profile Extraction for each micromovement (the number of samples is 100 in our Our goal is to match the test phase profile set against implementation)and put the phase profiles of each antenna- multiple template phase profile sets corresponding to different tag pair into a different set.We calibrate each set of phase micromovements,and determine which type of micromove- profiles to have consistent initial phase values,then we use ment the human subject is performing.Therefore,it is essential Dynamic Time Warping(DTW)to align these phase profiles to extract phase profiles from the RF signals while the human by normalizing the phase profiles along the time dimension. subject is performing multiple finger-based micromovements. Then,we calculate the average value and standard deviation of 1)Movement Segmentation:In practice,the human subject the specified set of phase profiles.Since there are 3 antennas, may continuously perform a series of finger movements. for each type of finger micromovement,we obtain a set of Therefore,the recognition system should first split these series 3×5=15 phase profiles. of movements into separate micromovement,then we can fur- Phase Profile Filtering:We observe that when the human ther identify which micromovement pattern the current finger subject is performing the multi-finger micromovement,the movement belongs to.As the human subject usually have a fingers can be divided into two categories,i.e.,the moving fin- short pause between two adjacent finger micromovements,RF- gers and the static fingers.The moving fingers usually lead to Glove uses phase changes to detect the start and end of a obvious variations in the phase profiles,thus the corresponding micromovement.Specifically,we leverage a sliding window phase profiles can be regarded as fingerprints to recognize the (which is set to 5s in our implementation)to continuously store micromovement.The static fingers usually lead to negligible the recent phase values of RF signals from multiple antenna- variations in the phase profiles,thus the corresponding phase tag pairs.For each antenna-tag pair,the system computes a profiles are not representative to recognize the micromove- derivative of these phase values,i.e.,the difference between ment.Therefore,the following two kinds of information can the current and the previous sample.When the difference from be used for micromovement recognition:1)The motion status one or more antenna-tag pairs exceeds a certain threshold,the of each finger,i.e.,we can label each finger as moving or system detects the beginning of a micromovement;similarly, static;2)the phase profiles of the moving fingers.Hence, when the difference from almost all antenna-tag pairs falls during the process of a specified micromovement,we can bellow the same threshold,the system detects the end of a compute the variance of the phase profile for each finger in the micromovement. corresponding time window.If the variance is greater than a 2)Test Phase Profiles:After segmentation,for the test threshold,then we label the corresponding finger as "moving", phase profile set,we use a matrix to store the phase values of otherwise,we label it as"static".After that,we further refer to the phase profiles.Similar to the phase matrix Rmxn.v,we the phase profiles of the moving finger as the template phase construct a matrix Pxn.to represent the test phase profiles. profile set for micromovement recognition. where each element P.;is defined as the phase value from Therefore,suppose there are m tags and n antennas (m= the tag Ti/n]received by antenna i%n at time point j.For 5 and n =3 in our implementation),the number of phase those missing values,we set them to a maximum value Umax samples is I',for each micromovement,we first use a vector M close to +oo for discrimination.Thus,each row of the matrix with length m to denote the motion status of each finger,where Pxn.!denotes the phase profile of each antenna-tag pair. we use“l”to denote the“noving”status and“O'to denote the "static"status of the finger.Then,for the template phase profile set,each set of phase profiles is pre-computed according to a C.Two-Phase Phase Profile Matching large number of samples of the specified micromovement.By In order to match the test phase profile to the template filtering out those phase profiles of static fingers which are phase profile,we first perform data calibration to align the not representative,we obtain the phase profiles of the moving phase profiles.Then,we leverage a two-phase approach for fingers as the template phase profiles.Hence,for the template matching.For the coarse-grained filtering,we leverage the phase profile set,we use a matrix to store the phase values of major variation trend of the phase profiles in the testing set the phase profiles.We construct a matrix Rmxn.to represent to filter out some template sets that the testing set should the template phase profiles,where each element Ri.;is defined not be matched to.For the fine-grained matching,we further as the phase value from the tag [i/n]received by antenna i%n match the testing set to one of the remaining template sets at time point j.Therefore,each row of the matrix Rmxn. by stretching or compressing the test phase profile to match denotes the phase profile of each antenna-tag pair.In addition, against multiple template phase profiles. we use a vector M'with length m x n as a mask to denote 1)Calibration:Note that even when the human subject which phase profile of antenna-tag pair is valid in the template performs the micromovement in slightly different positions, phase profile set.Fig.9 plots the template phase profile sets the initial phase values of the phase profiles are quite different. for five typical micromovements.We show the phase profiles To effectively recognize the micromovement,we only concern from antenna A,B and C,respectively.Specifically.for the the relative micromovement of multiple tags;therefore,the zoom in/out micromovements,since Tag 4 and Tag 5 label the initial phase values for multiple tags should be neglected. static fingers,we only plot the phase profiles of the moving When we match the test phase profile set against multiple fingers,i.e.,Tag I~3.For the rotate left/right,and flick,since template phase profile set,it is essential to first align these all tags label the moving fingers,we plot the phase profiles of phase profiles to reduce the initial phase deviation between all moving fingers,i.e.,Tag I~5. them.Our approach is as follows:for the phase profiles from
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 7 each micromovement as follows. We take multiple samples for each micromovement (the number of samples is 100 in our implementation) and put the phase profiles of each antennatag pair into a different set. We calibrate each set of phase profiles to have consistent initial phase values, then we use Dynamic Time Warping (DTW) to align these phase profiles by normalizing the phase profiles along the time dimension. Then, we calculate the average value and standard deviation of the specified set of phase profiles. Since there are 3 antennas, for each type of finger micromovement, we obtain a set of 3 × 5 = 15 phase profiles. Phase Profile Filtering: We observe that when the human subject is performing the multi-finger micromovement, the fingers can be divided into two categories, i.e., the moving fingers and the static fingers. The moving fingers usually lead to obvious variations in the phase profiles, thus the corresponding phase profiles can be regarded as fingerprints to recognize the micromovement. The static fingers usually lead to negligible variations in the phase profiles, thus the corresponding phase profiles are not representative to recognize the micromovement. Therefore, the following two kinds of information can be used for micromovement recognition: 1) The motion status of each finger, i.e., we can label each finger as moving or static; 2) the phase profiles of the moving fingers. Hence, during the process of a specified micromovement, we can compute the variance of the phase profile for each finger in the corresponding time window. If the variance is greater than a threshold, then we label the corresponding finger as “moving”, otherwise, we label it as “static”. After that, we further refer to the phase profiles of the moving finger as the template phase profile set for micromovement recognition. Therefore, suppose there are m tags and n antennas (m = 5 and n = 3 in our implementation), the number of phase samples is l , for each micromovement, we first use a vector M with length m to denote the motion status of each finger, where we use “1” to denote the “moving” status and “0” to denote the “static” status of the finger. Then, for the template phase profile set, each set of phase profiles is pre-computed according to a large number of samples of the specified micromovement. By filtering out those phase profiles of static fingers which are not representative, we obtain the phase profiles of the moving fingers as the template phase profiles. Hence, for the template phase profile set, we use a matrix to store the phase values of the phase profiles. We construct a matrix Rm×n,l to represent the template phase profiles, where each element Ri,j is defined as the phase value from the tag i/n received by antenna i%n at time point j. Therefore, each row of the matrix Rm×n,l denotes the phase profile of each antenna-tag pair. In addition, we use a vector M with length m × n as a mask to denote which phase profile of antenna-tag pair is valid in the template phase profile set. Fig. 9 plots the template phase profile sets for five typical micromovements. We show the phase profiles from antenna A, B and C, respectively. Specifically, for the zoom in/out micromovements, since Tag 4 and Tag 5 label the static fingers, we only plot the phase profiles of the moving fingers, i.e., Tag 1∼ 3. For the rotate left/right, and flick, since all tags label the moving fingers, we plot the phase profiles of all moving fingers, i.e., Tag 1∼ 5. B. Phase Profile Extraction Our goal is to match the test phase profile set against multiple template phase profile sets corresponding to different micromovements, and determine which type of micromovement the human subject is performing. Therefore, it is essential to extract phase profiles from the RF signals while the human subject is performing multiple finger-based micromovements. 1) Movement Segmentation: In practice, the human subject may continuously perform a series of finger movements. Therefore, the recognition system should first split these series of movements into separate micromovement, then we can further identify which micromovement pattern the current finger movement belongs to. As the human subject usually have a short pause between two adjacent finger micromovements, RFGlove uses phase changes to detect the start and end of a micromovement. Specifically, we leverage a sliding window (which is set to 5s in our implementation) to continuously store the recent phase values of RF signals from multiple antennatag pairs. For each antenna-tag pair, the system computes a derivative of these phase values, i.e., the difference between the current and the previous sample. When the difference from one or more antenna-tag pairs exceeds a certain threshold, the system detects the beginning of a micromovement; similarly, when the difference from almost all antenna-tag pairs falls bellow the same threshold, the system detects the end of a micromovement. 2) Test Phase Profiles: After segmentation, for the test phase profile set, we use a matrix to store the phase values of the phase profiles. Similar to the phase matrix Rm×n,l , we construct a matrix Pm×n,l to represent the test phase profiles, where each element Pi,j is defined as the phase value from the tag i/n received by antenna i%n at time point j. For those missing values, we set them to a maximum value vmax close to +∞ for discrimination. Thus, each row of the matrix Pm×n,l denotes the phase profile of each antenna-tag pair. C. Two-Phase Phase Profile Matching In order to match the test phase profile to the template phase profile, we first perform data calibration to align the phase profiles. Then, we leverage a two-phase approach for matching. For the coarse-grained filtering, we leverage the major variation trend of the phase profiles in the testing set to filter out some template sets that the testing set should not be matched to. For the fine-grained matching, we further match the testing set to one of the remaining template sets by stretching or compressing the test phase profile to match against multiple template phase profiles. 1) Calibration: Note that even when the human subject performs the micromovement in slightly different positions, the initial phase values of the phase profiles are quite different. To effectively recognize the micromovement, we only concern the relative micromovement of multiple tags; therefore, the initial phase values for multiple tags should be neglected. When we match the test phase profile set against multiple template phase profile set, it is essential to first align these phase profiles to reduce the initial phase deviation between them. Our approach is as follows: for the phase profiles from
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination. IEEE/ACM TRANSACTIONS ON NETWORKING Sonpling poin Sanpling poie (a) (b) (c) (d) (e) Fig.9.The phase profiles of the five micromovement in Fig.1.(a)Zoom in.(b)Zoom out.(c)Rotate left.(d)Rotate right.(e)Flick. one specified antenna,suppose the number of stable phase say 6,if 6t,or“-”ifd<-t.Then,we obtain a for both the test phase profiles and template phase profiles, state series like"++0--",we combine the adjacent states if we respectively calculate the average of their initial phase the belong to the same category and remove the outlier states values p and for the corresponding phase profiles,ie. between the consistent states;thus,we can get the series like 万=∑p and=∑1g,We then calibrte the test "+0-".Then,in regard to a test phase profile set,we just phase profiles by reducing Ap =p-for all the phase values match it against multiple template phase profile set in regard in test phase profiles.Thus,we can offset the initial phase to the variation trend.After this phase,we can filter out some deviation between the test phase profiles and the template template sets that the testing set should not be matched to, phase profiles. since the variation trend of phase profiles is a very obvious 2)Coarse-Grained Filtering With Curve Profile:For the metric to depict the micromovement. multiple finger-based micromovement,the motion statuses of 3)Fine-Grained Matching Via Time Warping:As different multiple fingers can be used as a coarse-grained filter for the human subjects may perform the same micromovement with micromovement recognition.Specifically,given the test phase different speeds,the phase profiles become stretched when the profile set,we can first compute its motion status vector M movement slows down and compressed when the movement and match it against the template vector M for multiple micro- speeds up.Besides,the test phase profiles usually have missing movements.If M,is matched to one or more template vectors, values,which makes traditional sequence matching algorithms we then further match it against the corresponding template such as [33]unreliable and impractical to tackle these prob- phase profile set.E.g.,this approach can effectively distinguish lems.Therefore,we need to stretch or compress the test the 2-finger micromovement and the 3-finger micromovement. phase profile to match against multiple template phase profiles Even if two fingers might have very similar moving patterns corresponding to different micromovements.To address the in both the two micromovements,we can still distinguish the above issues,we use the Dynamic Time Warping (DTW) two micromovements according to the moving status of the technique for efficient matching between the test phase profile third finger. set and the template phase profile set. Moreover,for the phase profiles of different micromovement In our approach,both the test phase profile set Pxn.and types as shown in Fig.9,we observe that the phase profiles for template phase profile set Rmxn contain a few temporal any antenna-tag pair all have a certain distinction degree in the sequences of phase values.These sequences comprise of phase curve profile,i.e.,the major variation trend of the phase value. sequences from different antennas and tags.Suppose there For example,as time goes by,some phase profiles keep stable are k(k<m x n)stable phase profiles in the specified while other phase profiles first increases and then gradually template phase profile set,then we can extract a vector from decreases.Hence.according to the variation trends of the each column of Rmxn.t to denote the relative position of phase profile,we can mainly classify them into three states: multiple tags at the specified time point.For example,we use 1)increase (+)2)decrease (-)and 3)keep stable (0).In this VR.j=(R,R)to represent the relative position of way,any phase profile can be described with the combination multiple tags at the jth(j<1')time point for the specified of the three states in terms of the variation trend,e.g.,for the template micromovement.Similarly,we use a vector Vp.i= micromovement of punch,the phase profile of antenna 2 and (P.P)to represent the relative position of multiple tag 4 pair can be depicted as state series"+0-"in regard to tags at the ith(i<)time point for the test micromovement. the variation trend.Accordingly,our coarse-grained filtering Therefore,for each pair of test phase profiles Pmxn.t and is performed as follows:for each phase profile,we compute template phase profiles Rmxn.,we can construct a distance a derivative of the phase values,i.e.,the difference between matrix Dixi as an input to the DTW algorithm,where each the current and the previous samples.Due to the existence element Di.;is defined as the Euclidean distance between each of slight fluctuations of phase values.we set a threshold t pair of phase profiles Vp.i and VR..Specifically,the Euclidean to differentiate it from the major variation trend.We use a distance can be calculated as follows:Dij=lVp.i-VR.jll. sliding window to continuously scan the phase changes.For The output of DTW is a warping path 0(01,...,}such the average phase changes within a specified time window w, that the distance y between the sequences is minimized:
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 8 IEEE/ACM TRANSACTIONS ON NETWORKING Fig. 9. The phase profiles of the five micromovement in Fig. 1. (a) Zoom in. (b) Zoom out. (c) Rotate left. (d) Rotate right. (e) Flick. one specified antenna, suppose the number of stable phase profiles in the specified template phase profiles is k, then, for both the test phase profiles and template phase profiles, we respectively calculate the average of their initial phase values pi and qi for the corresponding k phase profiles, ie, p¯ = k i=1 pi and q¯ = k i=1 qi. We then calibrate the test phase profiles by reducing Δp = ¯p−q¯ for all the phase values in test phase profiles. Thus, we can offset the initial phase deviation between the test phase profiles and the template phase profiles. 2) Coarse-Grained Filtering With Curve Profile: For the multiple finger-based micromovement, the motion statuses of multiple fingers can be used as a coarse-grained filter for the micromovement recognition. Specifically, given the test phase profile set, we can first compute its motion status vector Mt and match it against the template vector M for multiple micromovements. If Mt is matched to one or more template vectors, we then further match it against the corresponding template phase profile set. E.g., this approach can effectively distinguish the 2-finger micromovement and the 3-finger micromovement. Even if two fingers might have very similar moving patterns in both the two micromovements, we can still distinguish the two micromovements according to the moving status of the third finger. Moreover, for the phase profiles of different micromovement types as shown in Fig. 9, we observe that the phase profiles for any antenna-tag pair all have a certain distinction degree in the curve profile, i.e., the major variation trend of the phase value. For example, as time goes by, some phase profiles keep stable while other phase profiles first increases and then gradually decreases. Hence, according to the variation trends of the phase profile, we can mainly classify them into three states: 1) increase (+), 2) decrease (−), and 3) keep stable (0). In this way, any phase profile can be described with the combination of the three states in terms of the variation trend, e.g., for the micromovement of punch, the phase profile of antenna Z and tag 4 pair can be depicted as state series “+0−” in regard to the variation trend. Accordingly, our coarse-grained filtering is performed as follows: for each phase profile, we compute a derivative of the phase values, i.e., the difference between the current and the previous samples. Due to the existence of slight fluctuations of phase values, we set a threshold t to differentiate it from the major variation trend. We use a sliding window to continuously scan the phase changes. For the average phase changes within a specified time window w, say δ, if |δ| t, or “−” if δ < −t. Then, we obtain a state series like “++0−−”, we combine the adjacent states if the belong to the same category and remove the outlier states between the consistent states; thus, we can get the series like “+0−”. Then, in regard to a test phase profile set, we just match it against multiple template phase profile set in regard to the variation trend. After this phase, we can filter out some template sets that the testing set should not be matched to, since the variation trend of phase profiles is a very obvious metric to depict the micromovement. 3) Fine-Grained Matching Via Time Warping: As different human subjects may perform the same micromovement with different speeds, the phase profiles become stretched when the movement slows down and compressed when the movement speeds up. Besides, the test phase profiles usually have missing values, which makes traditional sequence matching algorithms such as [33] unreliable and impractical to tackle these problems. Therefore, we need to stretch or compress the test phase profile to match against multiple template phase profiles corresponding to different micromovements. To address the above issues, we use the Dynamic Time Warping (DTW) technique for efficient matching between the test phase profile set and the template phase profile set. In our approach, both the test phase profile set Pm×n,l and template phase profile set Rm×n,l contain a few temporal sequences of phase values. These sequences comprise of phase sequences from different antennas and tags. Suppose there are k(k ≤ m × n) stable phase profiles in the specified template phase profile set, then we can extract a vector from each column of Rm×n,l to denote the relative position of multiple tags at the specified time point. For example, we use VR,j = Rs1,j , ..., Rsk,j to represent the relative position of multiple tags at the jth(j<l ) time point for the specified template micromovement. Similarly, we use a vector VP,i = Ps1,i, ..., Psk,i to represent the relative position of multiple tags at the i th(i<l) time point for the test micromovement. Therefore, for each pair of test phase profiles Pm×n,l and template phase profiles Rm×n,l , we can construct a distance matrix Dl×l as an input to the DTW algorithm, where each element Di,j is defined as the Euclidean distance between each pair of phase profiles VP,i and VR,j . Specifically, the Euclidean distance can be calculated as follows: Di,j = VP,i − VR,j. The output of DTW is a warping path θ{θ1, ..., θk} such that the distance γ between the sequences is minimized:
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination. XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 0 60 170 170 160 150 0.8 140 0.6 130 130 120 120 0.4 110 110 100 100 100 .10 .100 100 (a) (b) (c) Fig.10.The contour lines of phases from RF-signals.(a)The ideal phase value.(b)The real phase value.(c)The difference between the ideal and real phase values. argmino D))Based on the above method. 0 ◆finger movement given the specified test phase profiles,we can enumerate phase change all possible template phase profiles and leverage DTW to compute the corresponding distance y.We then select the template phase profiles with the smallest distance,and set the corresponding micromovement as the recognition result Note that sometimes the human subject may unintentionally perform some finger micromovement which does not belong to any of the pre-defined micromovement patterns.In order to address this problem,when we use DTW to compute the distance y with different template phase profiles.we set a (b) threshold o.If all distances y with the pre-defined template phase profiles are larger than a,we then identify the current micromovement as irregular motions or unknown patterns. D.Reconstruct Template Phase Profiles Based on Location 1)Contour Lines of Phases From RF-Signals:According to Eq.(4),besides the diversity term u,the phase value only depends on the distance between the RFID antenna and the tag. Therefore,given the fixed position of the antenna,the contour lines of phases from the RF-signals mainly form concentric circles in front of the antenna in ideal situation.We conduct Jocation Pi a set of experiments in a two-dimensional space to validate (xiyi.zi) this hypothesis.We first build a two-dimensional coordinate (c) system,and deploy the antenna in the position (0,0).We Fig.11.Reconstruct template phase profiles.(a)2D model for the phase then plot the phase value in a rectangular space,which ranges variation and the finger movement.(b)3D model for the phase variation and from-100cm to 100cm in the X-axis,and ranges fromfrom the finger movement.(c)Angle of arrivals of the tag at different antennas. 100cm to 180cm in the Y-axis.Fig.10 shows the contour lines of phases from RF-signals.Specifically,Fig.10(a)and Fig.10(b)plot the ideal phase value and the real phase value, phase contour,i.e.,the concentric circles,to approximately respectively.We further plot the difference between the ideal depict the real phase contour. and real phase values.Suppose that for a specified position,the 2)Reconstruct Template Phase Profiles Based on Location: ideal phase value and real phase value are respectively and From the 2-dimensional view of the effective scanning area 0i,then the original difference is e =-0.Considering of an RFID antenna,the phase contour can be depicted as that these phase values have a period of 2m,we thus further concentric circles with the center at the antenna,as shown in compute the difference as follows: Fig.11(a).This actually forms a polar coordinate system with the origin O at the center of antenna.Then,for an arbitrary ec minfe,2n-eh. (14) micromovement starting from location Pi,we use the vector s to denote the finger movement,and use the vector I to denote Fig.10(c)plots the difference between the ideal and real phase the polar axis PO,which actually follows the gradient line values.We can observe that most of the differences are less of the phase contours.Besides,we use the angle ai to denote than 0.6,this means that the real phase distribution is close to the angle between the polar axis I and the finger movement s. the ideal phase distribution.Therefore,we can use the ideal As the moving distance of any micromovement,i.e.,s,is
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 9 Fig. 10. The contour lines of phases from RF-signals. (a) The ideal phase value. (b) The real phase value. (c) The difference between the ideal and real phase values. argminθ γθ = k i=1 Dx(θi),y(θi). Based on the above method, given the specified test phase profiles, we can enumerate all possible template phase profiles and leverage DTW to compute the corresponding distance γ. We then select the template phase profiles with the smallest distance, and set the corresponding micromovement as the recognition result. Note that sometimes the human subject may unintentionally perform some finger micromovement which does not belong to any of the pre-defined micromovement patterns. In order to address this problem, when we use DTW to compute the distance γ with different template phase profiles, we set a threshold α. If all distances γ with the pre-defined template phase profiles are larger than α, we then identify the current micromovement as irregular motions or unknown patterns. D. Reconstruct Template Phase Profiles Based on Location 1) Contour Lines of Phases From RF-Signals: According to Eq.(4), besides the diversity term μ, the phase value θ only depends on the distance between the RFID antenna and the tag. Therefore, given the fixed position of the antenna, the contour lines of phases from the RF-signals mainly form concentric circles in front of the antenna in ideal situation. We conduct a set of experiments in a two-dimensional space to validate this hypothesis. We first build a two-dimensional coordinate system, and deploy the antenna in the position (0, 0). We then plot the phase value in a rectangular space, which ranges from -100cm to 100cm in the X-axis, and ranges from from 100cm to 180cm in the Y-axis. Fig.10 shows the contour lines of phases from RF-signals. Specifically, Fig.10(a) and Fig.10(b) plot the ideal phase value and the real phase value, respectively. We further plot the difference between the ideal and real phase values. Suppose that for a specified position, the ideal phase value and real phase value are respectively θi and θi, then the original difference is e = |θi − θr|. Considering that these phase values have a period of 2π, we thus further compute the difference as follows: ec = min{e, 2π − e}. (14) Fig.10(c) plots the difference between the ideal and real phase values. We can observe that most of the differences are less than 0.6, this means that the real phase distribution is close to the ideal phase distribution. Therefore, we can use the ideal Fig. 11. Reconstruct template phase profiles. (a) 2D model for the phase variation and the finger movement. (b) 3D model for the phase variation and the finger movement. (c) Angle of arrivals of the tag at different antennas. phase contour, i.e., the concentric circles, to approximately depict the real phase contour. 2) Reconstruct Template Phase Profiles Based on Location: From the 2-dimensional view of the effective scanning area of an RFID antenna, the phase contour can be depicted as concentric circles with the center at the antenna, as shown in Fig.11(a). This actually forms a polar coordinate system with the origin O at the center of antenna. Then, for an arbitrary micromovement starting from location Pi, we use the vector s to denote the finger movement, and use the vector l to denote the polar axis PiO, which actually follows the gradient line of the phase contours. Besides, we use the angle αi to denote the angle between the polar axis l and the finger movement s. As the moving distance of any micromovement, i.e., s, is
This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination 10 IEEE/ACM TRANSACTIONS ON NETWORKING Original Measured Inferenced Distance Derived Template 150 1 0 Tag ID (a) (b) (c) (d) Fig.12.An example to reconstruct the template phase profiles based on location.(a)The original template phase profiles at a fixed position P1.(b)The measured template phase profiles at a different position P2.(c)The derived template phase profiles at the position P2.(d)The distance bewteen template phase profiles. smaller than half a wavelength 16.4cm,then,if we use As (1a:VA,21),(18:V8:218),and (ic:Vc,z1c)are A0i to denote the corresponding phase change,according to already known,we can compute the solution (s,ys,2s) the definition of phase. according to Eq.(19),and figure out the vector s =(s,ys,2s) 2π in the 3-dimensional space. △9，=入×2 cos i. (15) In this way,we can reconstruct the template phase profiles based on the exact location as follows:Let the human subject Similarly,from the 3-dimensional view,given the position perform a specified micromovement at a fixed position Pi.then P,the polar axis PO actually follows the gradient line of the corresponding phase profiles,i.e.,(t),0B(t)and ec(t). phase contours on the conical surface.Therefore,given an can be obtained for each snapshot t.Then,given the angle of arbitrary finger movement s,the actual phase variation should be linear to the projection of s on the gradient line 1,i.e., arrival from three antennas,i.e.,the polar axis lA,IB and lc, and the phase change△0a(t),△0B(t)and△fc(t),we can s cosai,as shown in Fig.11(b).According to the geometric use the above method to figure out the micromovement s(t) relationship,as a;is the angle between I and s,then at each snapshot t.Hence,in order to reconstruct the template 1.s phase profiles for a new position Pi,we can further figure Cos ai= (16 ·s out angle of arrival from three antennas,i.e..the polar axis I and I,and compute the corresponding phase change Hence,cos;s.According to Eq.(15).given the △fa(t),△fe(t)and△c(t)according to Eq.(19).Finally,. value of△0. we can derive the corresponding phase profiles (t),(t) △8·入1 and 0c(t). (17) Fig.12 shows an example to reconstruct the template phase profiles based on location.Without loss of generality,we illus- Sinceis a normalized vector of 1,given the fixed position trate the phase profiles collected from one specified antenna, P.can be computed as a constant vector (,z) e.g.,antenna A.We first obtain the template phase profiles at according to the relative position between Pi and O.Then, a fixed position P,as shown in Fig.12(a),then we move to a given the micromovement vector s=(s,ys,2),we have the different position P2 and obtain the measured template phase following linear relationship: profiles,as shown in Fig.12(b).Note that there exist obvious △0·入 difference between the above two template phase profiles.We 工1工g+1别g十a1之。= (18) further compute the derived template phase profiles at the 4π position P2,as shown in Fig.12(c).It can be observed that Therefore,for our RF-Glove system,since we deploy three the derived template phase profiles are very similar to the antennas on a flat plane in a mutually orthogonal manner,we measured template phase profiles at the position P2.We further can first perform 3D positioning to figure out the position of compute the distance between the original/derived template the specified tag,e.g.,Pi=(zi,yi,zi).Then,we can further phase profiles and the measured template phase profiles, figure out the angle of arrival of the tag from each antenna respectively,in Fig.12(d).Note that the distance is greatly A,B and C.i.e.,the polar axis lA,IB and lc.Hence,suppose reduced using our reconstructed template phase profiles than that a specified finger performs the micromovement s,we can the original template phase profiles. obtain the phase changes△fA,△0B and△9 c from the three antennas,respectively.According to Eq.(18), E.Discussions △0A·入 1)Scalability to Orientation Variation:When the human 工1A工8十1As十21A2g= subject is performing the micromovement,he/she may change △0B·入 the facing direction at different times.In this situation,the E1Bx8+1Bs十1B2g= (19) 4 orientation of the hands might be deviated from the one △0c·入 corresponding to the template phase profiles.This causes 工1cD8+1cs+21c28= 4π the testing phase profiles to be deviated from the template
This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. 10 IEEE/ACM TRANSACTIONS ON NETWORKING Fig. 12. An example to reconstruct the template phase profiles based on location. (a) The original template phase profiles at a fixed position P1. (b) The measured template phase profiles at a different position P2. (c) The derived template phase profiles at the position P2. (d) The distance bewteen template phase profiles. smaller than half a wavelength λ 2 ≈ 16.4cm, then, if we use Δθi to denote the corresponding phase change, according to the definition of phase, Δθi = 2π λ × 2s cos αi. (15) Similarly, from the 3-dimensional view, given the position Pi, the polar axis PiO actually follows the gradient line of phase contours on the conical surface. Therefore, given an arbitrary finger movement s, the actual phase variation should be linear to the projection of s on the gradient line l, i.e., s cosαi, as shown in Fig.11(b). According to the geometric relationship, as αi is the angle between l and s, then cos αi = l · s l·s . (16) Hence, cos αi · s = l l ·s. According to Eq. (15), given the value of Δθi, Δθi · λ 4π = l l · s. (17) Since l l is a normalized vector of l, given the fixed position Pi, l l can be computed as a constant vector xl, yl, zl according to the relative position between Pi and O. Then, given the micromovement vector s = xs, ys, zs , we have the following linear relationship: x1xs + y1ys + z1zs = Δθi · λ 4π . (18) Therefore, for our RF-Glove system, since we deploy three antennas on a flat plane in a mutually orthogonal manner, we can first perform 3D positioning to figure out the position of the specified tag, e.g., Pi = (xi, yi, zi). Then, we can further figure out the angle of arrival of the tag from each antenna A, B and C, i.e., the polar axis lA, lB and lC . Hence, suppose that a specified finger performs the micromovement s, we can obtain the phase changes ΔθA, ΔθB and ΔθC from the three antennas, respectively. According to Eq.(18), ⎧ ⎪⎪⎪⎪⎪⎪⎨ ⎪⎪⎪⎪⎪⎪⎩ x1A xs + y1A ys + z1A zs = ΔθA · λ 4π , x1B xs + y1B ys + z1B zs = ΔθB · λ 4π , x1C xs + y1C ys + z1C zs = ΔθC · λ 4π . (19) As (x1A , y1A , z1A ),(x1B , y1B , z1B ), and (x1C , y1C , z1C ) are already known, we can compute the solution (xs, ys, zs) according to Eq. (19), and figure out the vector s = xs, ys, zs in the 3-dimensional space. In this way, we can reconstruct the template phase profiles based on the exact location as follows: Let the human subject perform a specified micromovement at a fixed position Pi, then the corresponding phase profiles, i.e., θA(t), θB(t) and θC (t), can be obtained for each snapshot t. Then, given the angle of arrival from three antennas, i.e., the polar axis lA, lB and lC, and the phase change ΔθA(t), ΔθB(t) and ΔθC (t), we can use the above method to figure out the micromovement s(t) at each snapshot t. Hence, in order to reconstruct the template phase profiles for a new position Pj , we can further figure out angle of arrival from three antennas, i.e., the polar axis l A, l B and l C , and compute the corresponding phase change Δθ A(t), Δθ B(t) and Δθ C (t) according to Eq.(19). Finally, we can derive the corresponding phase profiles θ A(t), θ B(t) and θ C (t). Fig.12 shows an example to reconstruct the template phase profiles based on location. Without loss of generality, we illustrate the phase profiles collected from one specified antenna, e.g., antenna A. We first obtain the template phase profiles at a fixed position P1, as shown in Fig.12(a), then we move to a different position P2 and obtain the measured template phase profiles, as shown in Fig.12(b). Note that there exist obvious difference between the above two template phase profiles. We further compute the derived template phase profiles at the position P2, as shown in Fig.12(c). It can be observed that the derived template phase profiles are very similar to the measured template phase profiles at the position P2. We further compute the distance between the original/derived template phase profiles and the measured template phase profiles, respectively, in Fig.12(d). Note that the distance is greatly reduced using our reconstructed template phase profiles than the original template phase profiles. E. Discussions 1) Scalability to Orientation Variation: When the human subject is performing the micromovement, he/she may change the facing direction at different times. In this situation, the orientation of the hands might be deviated from the one corresponding to the template phase profiles. This causes the testing phase profiles to be deviated from the template