Signal Stream:S(t) Signal Preprocessing Data Calibration Semengtation CNN-based Training Model Reflection Feature 口A X (cm) Extraction (a)Reflections from tag array (b)Reflection RSS of tag array Finger Tracking Multi-touch Recognition Fig.5.Reflection model of a tag array NN-based Localization Correlation-based Image Construction represented as Sfree and Sreftect,respectively.Therefore,the 中 actual signal received by the reader can be represented as: Kalman-based Trace Smoothing CNN-based Gesture Sactual Sfree Sreflect. (3) Recognition Here,the finger movement affects Sreftect due to the change Fig.6.System framework of reflection path,thus both the RSS and phase of Sactual also Figure 5(b)illustrates the case where the finger is at (0,0)co- vary accordingly.In order to track the finger movements,we ordinate with 3cm height and each 1cmx lcm grid is supposed need to separate Srefteet from the received signals to roughly to deploy a tag.We set C to 1 for simplicity in this figure.We describe the distance of the reflection path.Specifically,we note that the power highly concentrates at the position of the can estimate Sreflect by subtracting Sactual by Sfree,where finger.Therefore,we can use the theoretical power distribution Sfree can be measured without the reflection object. as a pattern to estimate the finger position from the measured B.Impact of Finger Movement on a Tag Array power distribution of the whole tag array.By computing the The single tag model depicts the signal change on one tag theoretical power distribution in a fine-resolution manner,we caused by the finger movement,but the tag array involving are able to refine the recognition resolution of the tag array multiple tags,meaning the finger affects several adjacent tags with the correlation-based interpolation.In Section V-B,we at the same time.To better understand the reflected signals will show the effectiveness of the tag array model by extracting from the finger,we derive the theoretical model of tag array the refection feature from the reflection power distribution. as follows.In Figure 5(a),we use a one-dimension tag array to illustrate the finger impact on the tag array for simplicity. IV.SYSTEM OVERVIEW Specifically,the antenna A interrogates six tags 71 to 76,while A.System Architecture the finger H is hanging upon the tag array. The major objective of our work is to recognize the fine- According to the single tag model,we can derive the grained finger gestures via a device-free approach.Towards reflection feature Sreftect for each tag.Additionally,Sreftect this end,we design an RFID-based system,RF-finger,which can be further divided into two parts based on the reflection captures the signal changes on the tag array for gesture path in Figure 5(a): recognition.As shown in Figure 6,RF-finger consists of four Sreflect=SA→HSH→T: (4) main components:two core modules Signal Pre-processing where SAH represents the signal from A to H.SH and Reflection Feature Extraction,followed by two function- represents the signal reflected from H to 7i,and varies based ality modules Finger Tracking and Multi-touch Recognition. on the tag's position.In an ideal channel model [5].S Specifically,RF-finger takes as input the time-series signal is defined as: 1 si(t)received from each tag i of the tag array,including both SH-T=dT. (5) the RSSI and phase information.The Signal Pre-processing where diT is the distance between H and Ti.T is the module first calibrates the measured signal by interpolating the phase shift over the distance dFormally,the phase shift misreading signal and smoothing the signal.Next,we divide can be calculated from the wave length A as: the smoothed signals into separated gestures by analyzing the signal variance of all tags,which accurately estimates the 0HT4=2 dHT:mod 2n (6) 入 starting and ending point of a gesture.Then,the Reflection For each tag Ti,we can combine Eq.(4)and Eq.(5)to Feature Extraction module extracts the reflection features of calculate the power of the Srefteet [5]as: the gesture based on our reflection model in Section III. 1 Preflect =Sreflect2=C*- (7) After extracting the reflection features from RF signal,two main functionality modules are followed for finger tracking where denotes the module of the complex parameter to get and multi-touch gesture recognition.For the finger-writing, the power and C=SA2 is a constant power.Therefore,the Finger Tracking module locates the finger from the re- the magnitude of Preftect is determined by dHT,meaning flection features in each time stamp based on the K-Nearest the finger leads to larger reflection power to the close tags. Neighbors (KNN)algorithm.Locations in consecutive time Given the position of H,we can calculate the distribution stamps are connected together and smoothed via Kalman filter of reflection power Preftect in the 2D space from Eq.(7).to obtain a fine-grained trace.For the multi-touch gestures,O X Y A H T1 T2 T3 T4 T5 T6 H !H!Ti !A!H (a) Reflections from tag array X (cm) -5 0 5 Y (cm) -5 0 5 (b) Reflection RSS of tag array Fig. 5. Reflection model of a tag array. represented as Sf ree and Sref lect, respectively. Therefore, the actual signal received by the reader can be represented as: Sactual = Sf ree + Sref lect. (3) Here, the finger movement affects Sref lect due to the change of reflection path, thus both the RSS and phase of Sactual also vary accordingly. In order to track the finger movements, we need to separate Sref lect from the received signals to roughly describe the distance of the reflection path. Specifically, we can estimate Sref lect by subtracting Sactual by Sf ree, where Sf ree can be measured without the reflection object. B. Impact of Finger Movement on a Tag Array The single tag model depicts the signal change on one tag caused by the finger movement, but the tag array involving multiple tags, meaning the finger affects several adjacent tags at the same time. To better understand the reflected signals from the finger, we derive the theoretical model of tag array as follows. In Figure 5(a), we use a one-dimension tag array to illustrate the finger impact on the tag array for simplicity. Specifically, the antenna A interrogates six tags T1 to T6, while the finger H is hanging upon the tag array. According to the single tag model, we can derive the reflection feature Sref lect for each tag. Additionally, Sref lect can be further divided into two parts based on the reflection path in Figure 5(a): Sref lect = SA→HSH→Ti . (4) where SA→H represents the signal from A to H. SH→Ti represents the signal reflected from H to Ti , and varies based on the tag’s position. In an ideal channel model [5], SH→Ti is defined as: SH→Ti = 1 d 2 HT i e JθHT i , (5) where dHT i is the distance between H and Ti . θHT i is the phase shift over the distance dHT i . Formally, the phase shift can be calculated from the wave length λ as: θHT i = 2π dHT i λ mod 2π. (6) For each tag Ti , we can combine Eq. (4) and Eq. (5) to calculate the power of the Sref lect [5] as: Pref lect = |Sref lect| 2 = C ∗ 1 d 4 HT i , (7) where |·| denotes the module of the complex parameter to get the power and C = |SA→H| 2 is a constant power. Therefore, the magnitude of Pref lect is determined by dHT i , meaning the finger leads to larger reflection power to the close tags. Given the position of H, we can calculate the distribution of reflection power Pref lect in the 2D space from Eq. (7). Signal Stream: !"#$% Reflection Feature Extraction Finger Tracking KNN-based Localization Kalman-based Trace Smoothing Signal Preprocessing Data Calibration Semengtation Multi-touch Recognition Correlation-based Image Construction CNN-based Gesture Recognition CNN-based Training Model Fig. 6. System framework. Figure 5(b) illustrates the case where the finger is at (0, 0) co￾ordinate with 3cm height and each 1cm×1cm grid is supposed to deploy a tag. We set C to 1 for simplicity in this figure. We note that the power highly concentrates at the position of the finger. Therefore, we can use the theoretical power distribution as a pattern to estimate the finger position from the measured power distribution of the whole tag array. By computing the theoretical power distribution in a fine-resolution manner, we are able to refine the recognition resolution of the tag array with the correlation-based interpolation. In Section V-B, we will show the effectiveness of the tag array model by extracting the reflection feature from the reflection power distribution. IV. SYSTEM OVERVIEW A. System Architecture The major objective of our work is to recognize the fine￾grained finger gestures via a device-free approach. Towards this end, we design an RFID-based system, RF-finger, which captures the signal changes on the tag array for gesture recognition. As shown in Figure 6, RF-finger consists of four main components: two core modules Signal Pre-processing and Reflection Feature Extraction, followed by two function￾ality modules Finger Tracking and Multi-touch Recognition. Specifically, RF-finger takes as input the time-series signal si(t) received from each tag i of the tag array, including both the RSSI and phase information. The Signal Pre-processing module first calibrates the measured signal by interpolating the misreading signal and smoothing the signal. Next, we divide the smoothed signals into separated gestures by analyzing the signal variance of all tags, which accurately estimates the starting and ending point of a gesture. Then, the Reflection Feature Extraction module extracts the reflection features of the gesture based on our reflection model in Section III. After extracting the reflection features from RF signal, two main functionality modules are followed for finger tracking and multi-touch gesture recognition. For the finger-writing, the Finger Tracking module locates the finger from the re- flection features in each time stamp based on the K-Nearest Neighbors (KNN) algorithm. Locations in consecutive time stamps are connected together and smoothed via Kalman filter to obtain a fine-grained trace. For the multi-touch gestures, 4