This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS when the human subject performs the micromovement at the are attached to a controlling ball to detect the motions of ball positions out of the central beams of the antennas,the phase rotation from users.Compared with our RF-Glove system, profiles for the same micromovement might be different to a the tags in Tagball follow the same movement trace where certain extent at different positions.Hence,it is inaccurate to the tags in RF-Glove may follow different movement traces. directly match the testing set of phase profiles to the original RF-IDraw [10]uses a 2-dimensional array of RFID antennas template set of phase profiles.To address this challenge,we to track the movement trajectory of one finger attached with propose a solution to reconstruct the template phase profiles an RFID tag,so that it can reconstruct the trajectory shape based on the exact locations.We first propose a 3D positioning of the specified finger.However,RF-IDraw is designed to method based on the AoA method to figure out the locations track a fairly large range movement of one finger,e.g.,in of multiple fingers.Based on the fingers'location,we propose the size of 20~30cm.It does not work well for tracking the a model to depict the relationship between the phase variation concurrent movements of multiple fingers because its median and the specified movement.We further derive the correspond- accuracy is 3.7cm,which means that the accuracy of tracking ing template phase profiles based on the exact locations. two fingers could be 7.4cm,but finger movements are typically We make four key contributions in this paper.First,we 2cm to 5cm.Furthermore,the deployment cost of RF-IDraw propose RF-Glove,an RF signal based concurrent micromove- is relatively expensive as it requires an antenna array of ment recognition system.Second,we propose a 3D position- eight antennas and two RFID readers.Different from the ing model and a RF-micromovement model,respectively,to positioning-based techniques from RF-IDraw,in this paper,to depict the relationship between the multi-finger movement and achieve more accurate performance in micromovement recog- the RF-signals.Third,we propose a phase profiling based nition,we directly investigate the phase variation pattern from approach to RF signal based multi-finger micromovement the concurrent micromovement of multiple fingers,instead of recognition.Last,we implemented RF-Glove using COTS capturing the location variation of multiple fingers,since the RFID systems and evaluated its performance in realistic set- former metric captures the micromovement in much more fine tings.Experiment results show that we achieve an average granularity than the latter. accuracy of 92.1%under various moving speeds,orientation deviations.etc. III.MODELING RF SIGNAL VARIATIONS AND MULTI-FINGER MICROMOVEMENTS II.RELATED WORK Like the functionalities of the general purpose touch pad, the scheme of"multi-touch in the air"should also have both RFID-Based Localization:Prior work on RFID-based local- the positioning and gesture-recognition functionalities.The ization primarily rely on RSSI (Received Signal Strength) positioning functionality aims to locate the tagged fingers information [13],[14]to acquire the absolute location of an in a coarse-grained manner.In this way,we are able to object.State-of-the-art systems use phase value to estimate the easily recognize the large-range movement of the tagged absolute location of an object with higher accuracy [11],[12], fingers caused by the arm movement.The gesture-recognition [15]-[21].By deploying multiple antennas and measuring the functionality aims to further recognize the micromovement of phase difference between the received signals at different multiple tagged fingers in a fine-grained manner. antennas,these systems can effectively reduce the localization Therefore,to understand how RF-signals vary with large- error to a few centimeters.Further,PinIt exploits multi-path range movement,we propose a 3D positioning model that effect to accurately locate RFIDs by using synthetic aperture quantifies the relationship between the RF signal and the radar created via antenna motion to extract multi-path profiles position of tagged fingers in the 3-dimensional space.To for accurate localization [11].Tagoram exploits tag mobility to understand how RF signals vary with multi-finger micro- build a virtual antenna array,and uses differential augmented movements,we propose an RF micromovement model that hologram to facilitate the instant tracking of a mobile RFID quantifies the relationship between RF signals and multi-finger tag [12].While the above work mainly focuses on absolute micromovements.It shows that each different type of multi- object localization.Spatial-Temporal Phase Profiling (STPP) finger micromovements can be characterized by different RF is proposed for the relative localization of RFID tags [22]. phase variation patterns.Thus,by capturing the distinguishing Liu et al.[20]propose a pose sensing system called Tag- RF phase variation patterns,we can recognize different multi- Compass that uses a single tag to determine the orientation finger micromovements. as well as the position of the associated object.A completely We use the commercial RFID reader ImpinJ R420 and Laird different method based on the polarization properties of the S9028 antenna to receive RF signals.Laird S9028 antenna pro- RF waves is exploited to achieve fine-grained pose sensing. vides a consistent and continuous reading zone with circular RFID-Based Motion Tracking:Prior activity sensing sys- polarization.As shown in Figure 2(a),we deploy three anten- tems propose various approaches to recognize gestures for nas on the room ceiling,say A,B and C.The antenna pair activity sensing.These systems can be primarily classified into AB and AC are deployed in a mutually orthogonal fashion vision-based,infrared-based,electric field-based and wearable along the X-axis and Y-axis,respectively.By leveraging the approaches [23]-[25].RFID systems have recently been used Angle of Arrival (AoA)positioning method,the AoA from for trajectory tracking [10],[26].[27]and motion tracking the antenna pair AB can differentiate the movement along the [28]-[32].Lin et al.[29]proposed a 3D human-computer X-axis,while the AoA from the antenna pair AC can differ- interaction system called Tagball,where multiple passive tags entiate the movement along the Y-axis.We use Alien 9640This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 3 when the human subject performs the micromovement at the positions out of the central beams of the antennas, the phase profiles for the same micromovement might be different to a certain extent at different positions. Hence, it is inaccurate to directly match the testing set of phase profiles to the original template set of phase profiles. To address this challenge, we propose a solution to reconstruct the template phase profiles based on the exact locations. We first propose a 3D positioning method based on the AoA method to figure out the locations of multiple fingers. Based on the fingers’ location, we propose a model to depict the relationship between the phase variation and the specified movement. We further derive the correspond￾ing template phase profiles based on the exact locations. We make four key contributions in this paper. First, we propose RF-Glove, an RF signal based concurrent micromove￾ment recognition system. Second, we propose a 3D position￾ing model and a RF-micromovement model, respectively, to depict the relationship between the multi-finger movement and the RF-signals. Third, we propose a phase profiling based approach to RF signal based multi-finger micromovement recognition. Last, we implemented RF-Glove using COTS RFID systems and evaluated its performance in realistic set￾tings. Experiment results show that we achieve an average accuracy of 92.1% under various moving speeds, orientation deviations, etc. II. RELATED WORK RFID-Based Localization: Prior work on RFID-based local￾ization primarily rely on RSSI (Received Signal Strength) information [13], [14] to acquire the absolute location of an object. State-of-the-art systems use phase value to estimate the absolute location of an object with higher accuracy [11], [12], [15]–[21]. By deploying multiple antennas and measuring the phase difference between the received signals at different antennas, these systems can effectively reduce the localization error to a few centimeters. Further, PinIt exploits multi-path effect to accurately locate RFIDs by using synthetic aperture radar created via antenna motion to extract multi-path profiles for accurate localization [11]. Tagoram exploits tag mobility to build a virtual antenna array, and uses differential augmented hologram to facilitate the instant tracking of a mobile RFID tag [12]. While the above work mainly focuses on absolute object localization, Spatial-Temporal Phase Profiling (STPP) is proposed for the relative localization of RFID tags [22]. Liu et al. [20] propose a pose sensing system called Tag￾Compass that uses a single tag to determine the orientation as well as the position of the associated object. A completely different method based on the polarization properties of the RF waves is exploited to achieve fine-grained pose sensing. RFID-Based Motion Tracking: Prior activity sensing sys￾tems propose various approaches to recognize gestures for activity sensing. These systems can be primarily classified into vision-based, infrared-based, electric field-based and wearable approaches [23]–[25]. RFID systems have recently been used for trajectory tracking [10], [26], [27] and motion tracking [28]–[32]. Lin et al. [29] proposed a 3D human-computer interaction system called Tagball, where multiple passive tags are attached to a controlling ball to detect the motions of ball rotation from users. Compared with our RF-Glove system, the tags in Tagball follow the same movement trace where the tags in RF-Glove may follow different movement traces. RF-IDraw [10] uses a 2-dimensional array of RFID antennas to track the movement trajectory of one finger attached with an RFID tag, so that it can reconstruct the trajectory shape of the specified finger. However, RF-IDraw is designed to track a fairly large range movement of one finger, e.g., in the size of 20∼30cm. It does not work well for tracking the concurrent movements of multiple fingers because its median accuracy is 3.7cm, which means that the accuracy of tracking two fingers could be 7.4cm, but finger movements are typically 2cm to 5cm. Furthermore, the deployment cost of RF-IDraw is relatively expensive as it requires an antenna array of eight antennas and two RFID readers. Different from the positioning-based techniques from RF-IDraw, in this paper, to achieve more accurate performance in micromovement recog￾nition, we directly investigate the phase variation pattern from the concurrent micromovement of multiple fingers, instead of capturing the location variation of multiple fingers, since the former metric captures the micromovement in much more fine granularity than the latter. III. MODELING RF SIGNAL VARIATIONS AND MULTI-FINGER MICROMOVEMENTS Like the functionalities of the general purpose touch pad, the scheme of “multi-touch in the air” should also have both the positioning and gesture-recognition functionalities. The positioning functionality aims to locate the tagged fingers in a coarse-grained manner. In this way, we are able to easily recognize the large-range movement of the tagged fingers caused by the arm movement. The gesture-recognition functionality aims to further recognize the micromovement of multiple tagged fingers in a fine-grained manner. Therefore, to understand how RF-signals vary with large￾range movement, we propose a 3D positioning model that quantifies the relationship between the RF signal and the position of tagged fingers in the 3-dimensional space. To understand how RF signals vary with multi-finger micro￾movements, we propose an RF micromovement model that quantifies the relationship between RF signals and multi-finger micromovements. It shows that each different type of multi- finger micromovements can be characterized by different RF phase variation patterns. Thus, by capturing the distinguishing RF phase variation patterns, we can recognize different multi- finger micromovements. We use the commercial RFID reader ImpinJ R420 and Laird S9028 antenna to receive RF signals. Laird S9028 antenna pro￾vides a consistent and continuous reading zone with circular polarization. As shown in Figure 2(a), we deploy three anten￾nas on the room ceiling, say A, B and C. The antenna pair AB and AC are deployed in a mutually orthogonal fashion along the X-axis and Y -axis, respectively. By leveraging the Angle of Arrival (AoA) positioning method, the AoA from the antenna pair AB can differentiate the movement along the X-axis, while the AoA from the antenna pair AC can differ￾entiate the movement along the Y -axis. We use Alien 9640