2153020532510540507 2.4 Gesture- 2 49.050851051950050050.0 22 Release >3510500500-4904992051.0 (uerp) -497500500490480500486 Threshold ● 502508-49050050050051.0 riginal phas 14 ● 23 587 After interpolating smoothing 2 150 200 30 (a)Shuffled deployment (b)RSSI distribution of shuffled 50 100 100 200 Sample index Sample index deployment (a)Signal calibration (b)Gesture segmentation Fig.7.Shuffled deployment of the dense tag array. Fig.8.Illustration of signal preprocessing in RF-finger. the Multi-touch Recognition module leverages a Convolu- 4)=86-4)+(06+)-66-1》-车1 (8) tional Neural Network(CNN)to automatically classify each tit1-ti-1 gesture from the visual features.Particularly,it constructs a where 0(ti_1)and 0(ti1)are two adjacent phase readings before and after time ti.After interpolation,a moving average 3-frame image of the gesture from the reflection features, which describes the influence range of the multiple fingers filter is applied to smooth the signal,which further removes high-frequency noise.Figure 8(a)illustrates the effectiveness in the starting/middle/ending period of the gestures.Then we learn the neural network model from the 3-frame image for of our data calibration by comparing the phase stream before gesture classification.Finally,we can recognize the gestures and after data calibration.The phase stream shown in the figure is from one tag in the array,when the user is performing by analyzing the classification scores based on CNN right rotate gesture.From the enlarged figure,we could clearly B.Dense Tag Array Deployment see the misreadings are well interpolated.Moreover,after As illustrated in Section II-A,we observe that the adjacent smoothing,the high-frequency serrated waves are removed. tags in the dense tag array have great impacts on the signal To capture the signal pattern of a specific finger movement, quality of other tags due to the electromagnetic interference [8, we need to identify its starting and ending point,which 19].The principle behind such influence is the electromagnetic correspond to the gesture people tend to raise the hand up interference between the two tags [8].As a result,the parallel and drop the hand down.Therefore,a segmentation method deployed tags will affect the nearby tags due to the mutual based on the detection of the calibrated RF-signals variance interference.To eliminate such mutual interference,we shuffle is developed to detect the actions of rasing/releasing hand the directions of part tags as shown in Figure 7(a)by making to segment gestures.Intuitively,we observe that the signal the nearby tags perpendicular to each other.In this way,we should be stable when people drop the hand down,and the can minimize the interference between nearby tags by making signal of some tags experiences distinct variations when the the electromagnetic interference perpendicular to each tag.As user performs the gestures.Therefore,we further leverage a a result,we can then achieve a stable RSSI measurement sliding window to calculate the variance stream of each tag across all tags,which is shown in Figure 7(b).Therefore,we from the calibrated RF-signals,and the starting/ending points adopt the perpendicular deployment of the tag array in our should have large variance values.Figure 8(b)illustrates the system. variance stream of all the 35 tags,which takes as input the V.RF-FINGER SYSTEM DESIGN calibrated phase stream.We find only part of the tags have In this section,we will talk about the detailed design of the large signal variance at the same time,because the finger only proposed RF-finger system.Specifically,we first preprocess affects several tags close to the finger.Thus,we continuously the raw RF-signals and then extract the reflection features to calculate the maximum variance of each sliding window for depict the finger influence on the tag array.Finally,we track the maximum variance stream.Based on the first and last the finger trace and recognize the multi-touch gestures from peak of the max variance stream,we can detect the action of these reflection features. raising/releasing hand and then take the signal stream between them as the gesture signal. A.Signal Preprocessing Given the received RF-signals,which involve some inherent B.Refection Feature Extraction measurement defects such as misreading tags and noise,the After signal processing,we have the segmented and noise- data calibration process is developed to improve the reliability less signal of each individual gesture,so we first leverage of the RF-signals by interpolating the misreading tags and the reflection model in Section III-A to derive the reflection smoothing the signal.In RFID system,the misreading tags signals Sreftect of each tag.Then we extract the reflection are usually caused by the highly dynamic environment during features from the Sreftect as the likelihood distribution inside the finger movement.Therefore,we can interpolate the mis-the tag array zone,where the likelihood of each position reading RF-signals from adjacent sampling rounds based on depicts the probability that the finger locates at the position. the continuous movement of the finger.Take a phase stream Before defining the likelihood.we derive the reflection signal 0(t)as an example,which is time-series phase values from of each tag by removing the free-space signal as Srefiect one tag.If there is a misreading phase (ti),we calculate the Sactual-Sfree.Particularly,Sactual is collected during the interpolation value from other phase reading as: gesture period and Sfree is collected before the gesture. 5(a) Shuffled deployment -52.7 -49.0 -51.0 -49.7 -50.2 -53.0 -50.6 -50.0 -50.0 -50.6 -52.0 -51.0 -50.0 -50.0 -49.0 -53.2 -51.9 -49.0 -49.0 -50.0 -51.0 -50.0 -49.9 -48.0 -50.0 -54.0 -50.0 -52.0 -50.0 -50.0 -50.7 -50.0 -51.0 -48.6 -51.0 X 1 2 3 4 5 6 7 Y 1 2 3 4 5 (b) RSSI distribution of shuffled deployment Fig. 7. Shuffled deployment of the dense tag array. the Multi-touch Recognition module leverages a Convolu￾tional Neural Network (CNN) to automatically classify each gesture from the visual features. Particularly, it constructs a 3-frame image of the gesture from the reflection features, which describes the influence range of the multiple fingers in the starting/middle/ending period of the gestures. Then we learn the neural network model from the 3-frame image for gesture classification. Finally, we can recognize the gestures by analyzing the classification scores based on CNN. B. Dense Tag Array Deployment As illustrated in Section II-A, we observe that the adjacent tags in the dense tag array have great impacts on the signal quality of other tags due to the electromagnetic interference [8, 19]. The principle behind such influence is the electromagnetic interference between the two tags [8]. As a result, the parallel deployed tags will affect the nearby tags due to the mutual interference. To eliminate such mutual interference, we shuffle the directions of part tags as shown in Figure 7(a) by making the nearby tags perpendicular to each other. In this way, we can minimize the interference between nearby tags by making the electromagnetic interference perpendicular to each tag. As a result, we can then achieve a stable RSSI measurement across all tags, which is shown in Figure 7(b). Therefore, we adopt the perpendicular deployment of the tag array in our system. V. RF-FINGER SYSTEM DESIGN In this section, we will talk about the detailed design of the proposed RF-finger system. Specifically, we first preprocess the raw RF-signals and then extract the reflection features to depict the finger influence on the tag array. Finally, we track the finger trace and recognize the multi-touch gestures from these reflection features. A. Signal Preprocessing Given the received RF-signals, which involve some inherent measurement defects such as misreading tags and noise, the data calibration process is developed to improve the reliability of the RF-signals by interpolating the misreading tags and smoothing the signal. In RFID system, the misreading tags are usually caused by the highly dynamic environment during the finger movement. Therefore, we can interpolate the mis￾reading RF-signals from adjacent sampling rounds based on the continuous movement of the finger. Take a phase stream θ(t) as an example, which is time-series phase values from one tag. If there is a misreading phase θ(ti), we calculate the interpolation value from other phase reading as: Sample index 0 50 100 150 200 Phase (radian) 1.2 1.4 1.6 1.8 2 2.2 2.4 Original phase After interpolating After interpolating & smoothing 2 2.1 2.2 2.3 (a) Signal calibration Sample index 0 100 200 300 Phase (radian) 0 1 2 3 4 Variance of one tag Max variance of all tags Threshold Gesture Raise hand Release hand (b) Gesture segmentation Fig. 8. Illustration of signal preprocessing in RF-finger. ˆθ(ti) = θ(ti−1) + (θ(ti+1) − θ(ti−1)) ti − ti−1 ti+1 − ti−1 , (8) where θ(ti−1) and θ(ti+1) are two adjacent phase readings before and after time ti . After interpolation, a moving average filter is applied to smooth the signal, which further removes high-frequency noise. Figure 8(a) illustrates the effectiveness of our data calibration by comparing the phase stream before and after data calibration. The phase stream shown in the figure is from one tag in the array, when the user is performing right rotate gesture. From the enlarged figure, we could clearly see the misreadings are well interpolated. Moreover, after smoothing, the high-frequency serrated waves are removed. To capture the signal pattern of a specific finger movement, we need to identify its starting and ending point, which correspond to the gesture people tend to raise the hand up and drop the hand down. Therefore, a segmentation method based on the detection of the calibrated RF-signals variance is developed to detect the actions of rasing/releasing hand to segment gestures. Intuitively, we observe that the signal should be stable when people drop the hand down, and the signal of some tags experiences distinct variations when the user performs the gestures. Therefore, we further leverage a sliding window to calculate the variance stream of each tag from the calibrated RF-signals, and the starting/ending points should have large variance values. Figure 8(b) illustrates the variance stream of all the 35 tags, which takes as input the calibrated phase stream. We find only part of the tags have large signal variance at the same time, because the finger only affects several tags close to the finger. Thus, we continuously calculate the maximum variance of each sliding window for the maximum variance stream. Based on the first and last peak of the max variance stream, we can detect the action of raising/releasing hand and then take the signal stream between them as the gesture signal. B. Reflection Feature Extraction After signal processing, we have the segmented and noise￾less signal of each individual gesture, so we first leverage the reflection model in Section III-A to derive the reflection signals Sref lect of each tag. Then we extract the reflection features from the Sref lect as the likelihood distribution inside the tag array zone, where the likelihood of each position depicts the probability that the finger locates at the position. Before defining the likelihood, we derive the reflection signal of each tag by removing the free-space signal as Sref lect = Sactual − Sf ree. Particularly, Sactual is collected during the gesture period and Sf ree is collected before the gesture. 5