This article has been accepted for inclusion in a future issue of this journal.Content is final as presented,with the exception of pagination XIE et al:MULTI-TOUCH IN THE AIR:CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS each micromovement as follows.We take multiple samples B.Phase Profile Extraction for each micromovement (the number of samples is 100 in our Our goal is to match the test phase profile set against implementation)and put the phase profiles of each antenna- multiple template phase profile sets corresponding to different tag pair into a different set.We calibrate each set of phase micromovements,and determine which type of micromove- profiles to have consistent initial phase values,then we use ment the human subject is performing.Therefore,it is essential Dynamic Time Warping(DTW)to align these phase profiles to extract phase profiles from the RF signals while the human by normalizing the phase profiles along the time dimension. subject is performing multiple finger-based micromovements. Then,we calculate the average value and standard deviation of 1)Movement Segmentation:In practice,the human subject the specified set of phase profiles.Since there are 3 antennas, may continuously perform a series of finger movements. for each type of finger micromovement,we obtain a set of Therefore,the recognition system should first split these series 3×5=15 phase profiles. of movements into separate micromovement,then we can fur- Phase Profile Filtering:We observe that when the human ther identify which micromovement pattern the current finger subject is performing the multi-finger micromovement,the movement belongs to.As the human subject usually have a fingers can be divided into two categories,i.e.,the moving fin- short pause between two adjacent finger micromovements,RF- gers and the static fingers.The moving fingers usually lead to Glove uses phase changes to detect the start and end of a obvious variations in the phase profiles,thus the corresponding micromovement.Specifically,we leverage a sliding window phase profiles can be regarded as fingerprints to recognize the (which is set to 5s in our implementation)to continuously store micromovement.The static fingers usually lead to negligible the recent phase values of RF signals from multiple antenna- variations in the phase profiles,thus the corresponding phase tag pairs.For each antenna-tag pair,the system computes a profiles are not representative to recognize the micromove- derivative of these phase values,i.e.,the difference between ment.Therefore,the following two kinds of information can the current and the previous sample.When the difference from be used for micromovement recognition:1)The motion status one or more antenna-tag pairs exceeds a certain threshold,the of each finger,i.e.,we can label each finger as moving or system detects the beginning of a micromovement;similarly, static;2)the phase profiles of the moving fingers.Hence, when the difference from almost all antenna-tag pairs falls during the process of a specified micromovement,we can bellow the same threshold,the system detects the end of a compute the variance of the phase profile for each finger in the micromovement. corresponding time window.If the variance is greater than a 2)Test Phase Profiles:After segmentation,for the test threshold,then we label the corresponding finger as "moving", phase profile set,we use a matrix to store the phase values of otherwise,we label it as"static".After that,we further refer to the phase profiles.Similar to the phase matrix Rmxn.v,we the phase profiles of the moving finger as the template phase construct a matrix Pxn.to represent the test phase profiles. profile set for micromovement recognition. where each element P.;is defined as the phase value from Therefore,suppose there are m tags and n antennas (m= the tag Ti/n]received by antenna i%n at time point j.For 5 and n =3 in our implementation),the number of phase those missing values,we set them to a maximum value Umax samples is I',for each micromovement,we first use a vector M close to +oo for discrimination.Thus,each row of the matrix with length m to denote the motion status of each finger,where Pxn.!denotes the phase profile of each antenna-tag pair. we use“l”to denote the“noving”status and“O'to denote the "static"status of the finger.Then,for the template phase profile set,each set of phase profiles is pre-computed according to a C.Two-Phase Phase Profile Matching large number of samples of the specified micromovement.By In order to match the test phase profile to the template filtering out those phase profiles of static fingers which are phase profile,we first perform data calibration to align the not representative,we obtain the phase profiles of the moving phase profiles.Then,we leverage a two-phase approach for fingers as the template phase profiles.Hence,for the template matching.For the coarse-grained filtering,we leverage the phase profile set,we use a matrix to store the phase values of major variation trend of the phase profiles in the testing set the phase profiles.We construct a matrix Rmxn.to represent to filter out some template sets that the testing set should the template phase profiles,where each element Ri.;is defined not be matched to.For the fine-grained matching,we further as the phase value from the tag [i/n]received by antenna i%n match the testing set to one of the remaining template sets at time point j.Therefore,each row of the matrix Rmxn. by stretching or compressing the test phase profile to match denotes the phase profile of each antenna-tag pair.In addition, against multiple template phase profiles. we use a vector M'with length m x n as a mask to denote 1)Calibration:Note that even when the human subject which phase profile of antenna-tag pair is valid in the template performs the micromovement in slightly different positions, phase profile set.Fig.9 plots the template phase profile sets the initial phase values of the phase profiles are quite different. for five typical micromovements.We show the phase profiles To effectively recognize the micromovement,we only concern from antenna A,B and C,respectively.Specifically.for the the relative micromovement of multiple tags;therefore,the zoom in/out micromovements,since Tag 4 and Tag 5 label the initial phase values for multiple tags should be neglected. static fingers,we only plot the phase profiles of the moving When we match the test phase profile set against multiple fingers,i.e.,Tag I~3.For the rotate left/right,and flick,since template phase profile set,it is essential to first align these all tags label the moving fingers,we plot the phase profiles of phase profiles to reduce the initial phase deviation between all moving fingers,i.e.,Tag I~5. them.Our approach is as follows:for the phase profiles fromThis article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. XIE et al.: MULTI-TOUCH IN THE AIR: CONCURRENT MICROMOVEMENT RECOGNITION USING RF SIGNALS 7 each micromovement as follows. We take multiple samples for each micromovement (the number of samples is 100 in our implementation) and put the phase profiles of each antenna￾tag pair into a different set. We calibrate each set of phase profiles to have consistent initial phase values, then we use Dynamic Time Warping (DTW) to align these phase profiles by normalizing the phase profiles along the time dimension. Then, we calculate the average value and standard deviation of the specified set of phase profiles. Since there are 3 antennas, for each type of finger micromovement, we obtain a set of 3 × 5 = 15 phase profiles. Phase Profile Filtering: We observe that when the human subject is performing the multi-finger micromovement, the fingers can be divided into two categories, i.e., the moving fin￾gers and the static fingers. The moving fingers usually lead to obvious variations in the phase profiles, thus the corresponding phase profiles can be regarded as fingerprints to recognize the micromovement. The static fingers usually lead to negligible variations in the phase profiles, thus the corresponding phase profiles are not representative to recognize the micromove￾ment. Therefore, the following two kinds of information can be used for micromovement recognition: 1) The motion status of each finger, i.e., we can label each finger as moving or static; 2) the phase profiles of the moving fingers. Hence, during the process of a specified micromovement, we can compute the variance of the phase profile for each finger in the corresponding time window. If the variance is greater than a threshold, then we label the corresponding finger as “moving”, otherwise, we label it as “static”. After that, we further refer to the phase profiles of the moving finger as the template phase profile set for micromovement recognition. Therefore, suppose there are m tags and n antennas (m = 5 and n = 3 in our implementation), the number of phase samples is l , for each micromovement, we first use a vector M with length m to denote the motion status of each finger, where we use “1” to denote the “moving” status and “0” to denote the “static” status of the finger. Then, for the template phase profile set, each set of phase profiles is pre-computed according to a large number of samples of the specified micromovement. By filtering out those phase profiles of static fingers which are not representative, we obtain the phase profiles of the moving fingers as the template phase profiles. Hence, for the template phase profile set, we use a matrix to store the phase values of the phase profiles. We construct a matrix Rm×n,l to represent the template phase profiles, where each element Ri,j is defined as the phase value from the tag i/n received by antenna i%n at time point j. Therefore, each row of the matrix Rm×n,l denotes the phase profile of each antenna-tag pair. In addition, we use a vector M with length m × n as a mask to denote which phase profile of antenna-tag pair is valid in the template phase profile set. Fig. 9 plots the template phase profile sets for five typical micromovements. We show the phase profiles from antenna A, B and C, respectively. Specifically, for the zoom in/out micromovements, since Tag 4 and Tag 5 label the static fingers, we only plot the phase profiles of the moving fingers, i.e., Tag 1∼ 3. For the rotate left/right, and flick, since all tags label the moving fingers, we plot the phase profiles of all moving fingers, i.e., Tag 1∼ 5. B. Phase Profile Extraction Our goal is to match the test phase profile set against multiple template phase profile sets corresponding to different micromovements, and determine which type of micromove￾ment the human subject is performing. Therefore, it is essential to extract phase profiles from the RF signals while the human subject is performing multiple finger-based micromovements. 1) Movement Segmentation: In practice, the human subject may continuously perform a series of finger movements. Therefore, the recognition system should first split these series of movements into separate micromovement, then we can fur￾ther identify which micromovement pattern the current finger movement belongs to. As the human subject usually have a short pause between two adjacent finger micromovements, RF￾Glove uses phase changes to detect the start and end of a micromovement. Specifically, we leverage a sliding window (which is set to 5s in our implementation) to continuously store the recent phase values of RF signals from multiple antenna￾tag pairs. For each antenna-tag pair, the system computes a derivative of these phase values, i.e., the difference between the current and the previous sample. When the difference from one or more antenna-tag pairs exceeds a certain threshold, the system detects the beginning of a micromovement; similarly, when the difference from almost all antenna-tag pairs falls bellow the same threshold, the system detects the end of a micromovement. 2) Test Phase Profiles: After segmentation, for the test phase profile set, we use a matrix to store the phase values of the phase profiles. Similar to the phase matrix Rm×n,l , we construct a matrix Pm×n,l to represent the test phase profiles, where each element Pi,j is defined as the phase value from the tag i/n received by antenna i%n at time point j. For those missing values, we set them to a maximum value vmax close to +∞ for discrimination. Thus, each row of the matrix Pm×n,l denotes the phase profile of each antenna-tag pair. C. Two-Phase Phase Profile Matching In order to match the test phase profile to the template phase profile, we first perform data calibration to align the phase profiles. Then, we leverage a two-phase approach for matching. For the coarse-grained filtering, we leverage the major variation trend of the phase profiles in the testing set to filter out some template sets that the testing set should not be matched to. For the fine-grained matching, we further match the testing set to one of the remaining template sets by stretching or compressing the test phase profile to match against multiple template phase profiles. 1) Calibration: Note that even when the human subject performs the micromovement in slightly different positions, the initial phase values of the phase profiles are quite different. To effectively recognize the micromovement, we only concern the relative micromovement of multiple tags; therefore, the initial phase values for multiple tags should be neglected. When we match the test phase profile set against multiple template phase profile set, it is essential to first align these phase profiles to reduce the initial phase deviation between them. Our approach is as follows: for the phase profiles from