IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 12 7.1 Template Generation o-Click --Press and hold Swipe The template is generated in advance as the reference for the Click Press and hold touch estimation,especially for figuring out the real-time touch position according to the RSSI deviation.To collect the actual position of the finger on the tag,we utilize Op- tiTrack [33]to track the finger position as the ground-truth. The generation can be divided into three steps:re-sampling, Swipe feature extraction,and curve fitting.1)Re-sampling:By syn- 2U▣ Time(sec.) chronizing the finger trace from OptiTrack and the received (a)Operation illustration (b)RSSI variation RF-signals of touch tag,we are able to capture the ground- Fig.17.Different RSSI variation patterns of three gestures truth of the finger position on the tag.Note that,the two kinds of data have different sampling rates,i.e.,120Hz of time of the touch situation than "click".The threshold of the OptiTrack and around 100Hz of RF-signals,so we re-sample lasting period is set to be 0.7 seconds.While for swipe,the the data to the same sampling rate of 100Hz to correspond RSSI value decreases suddenly,then changes continuously RSSI values to each touch point.2)Feature Extraction:As for with different touch positions in several time intervals.If the a touch gesture,the touch gesture usually starts with the RSSI changes significantly over successive time intervals,it finger hangs over the tag,then the finger presses down to is identified as the swipe operation.Empirically,the time the tag.Considering this continuous process,it is not hard interval is set to be 0.1 seconds,the number of successive to take the RSSI value just before touching the tag as the time intervals is 3.Note that,the tag center is a fallible area reference value for calculating the RSSI deviation,we give where the absolute sudden RSSI variation is a bit small,it an example in Section 7.5.With the reference RSSI value,we is easy to mistake the click or press and hold for the noise. can extract the RSSI deviation by subtracting the reference According to the yellow swipe data in Fig.17(b),when the value from the received RSSI values.3)Curve Fitting:Apply finger leaves the tag from the tag center,the RSSI changes the spline fitting to the derive RSSI deviation pattern,so only about 2dB.It is because that when the touch position as to obtain the continuous relationship between the touch approaches the tag center,the conjugate mismatching condi- point and the RSSI deviation with the discrete RSSI values tion gets better,so the power loss gets less.Especially when at finite positions.Considering the uncertain noise during the finger just locates at the tag center,touching both the once swipe,a user is required to repeat the swipe across the two tag antennas,new conjugate matching will form,and tag from left to right three times,then we average the three the power loss could be less than 1dB.As the RSSI of a fitting curves to extract the RSSI deviation template. tag in the static environment will change about 1dB due to the ambient noise,a too small RSSI variation threshold will 7.2 Data Preprocessing for Touch Gesture Detection mistake the noise for the touch gesture,so if the range of To eliminate the data fluctuation due to the ambient noise in RSSI values in a short time interval exceeds 2dB,we think the environment,before performing the fine-grained touch the RSSI changes significantly and suddenly,it probably gesture detection,we first use the moving average filter to indicates the finger vertical movement or the swipe,instead smooth the data.Specifically,for each sample i,take the of the noise.The size of the fallible area is related to the RSSl average value of the nearest five samples,from i-2 to variation threshold,and it is about 1cm with our settings i+2,to replace sample i.Meanwhile,similar to the template Considering the width of the finger,the effective operation generation,we determine the reference RSSI value and then size on the used touch tag is around 8.5cm.Thankfully,the calculate the RSSI deviation for the touch position estima- size of this fallible area is not large,we suggest to avoid to tion.As we update the reference RSSI value for each touch design a button in this fallible area.Also,as depicted above, operation,we can eliminate the major difference of raw RSSI the RSSI deviation pattern is symmetric during the swipe values due to different positions or orientations.Therefore, across the whole tag,so in our design,we utilize the half the derived RSSI deviation mainly reflects the impedance part of the tag as the button interface and the other half as change caused by the finger touch on the tag's antenna,so the slider interface,an example is illustrated in Fig.29.This it is robust to different positions and orientations. may seem to be the limitation to our work,but in fact,such design is natural and friendly,e.g.,the touch bar on Apple 7.3 Gesture Detection MacBook Pro contains both buttons and the slider.How to Each touch gesture includes three periods:finger hanging extend the operation area of the tag and further improve the over the tag,finger touching on the tag,finger leaving interaction experience will be studied in our future work. the tag.Fig.17 plots the RSSI variations of different touch gestures,beginning with pressing down to the tag and 7.4 Touch Estimation ending with lifting the finger from the tag.Here,the swipe With the known touch interface design,as which part func- is performed on the half of the tag,i.e.,from left to center. tions as buttons and which part functions as the slider, It is found that the finger vertical movement,including we can estimate the absolute touch position on the tag by finger up and finger down,will bring the sudden RSSI comparing the extracted RSSI deviation with the template. variation due to the quick action.Specifically,for click,the For swipe,by combining the consecutive touch positions, RSSI decreases suddenly,then lasts for a short period and we are able to determine the swipe direction,and calculate recovers to original value suddenly.For press and hold,it the swipe distance as well as the swipe speed,which are the is similar to "click",the only difference is its longer lasting key factors for analyzing the friendly interaction commands.IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 12 7.1 Template Generation The template is generated in advance as the reference for the touch estimation, especially for figuring out the real-time touch position according to the RSSI deviation. To collect the actual position of the finger on the tag, we utilize Op￾tiTrack [33] to track the finger position as the ground-truth. The generation can be divided into three steps: re-sampling, feature extraction, and curve fitting. 1) Re-sampling: By syn￾chronizing the finger trace from OptiTrack and the received RF-signals of touch tag, we are able to capture the ground￾truth of the finger position on the tag. Note that, the two kinds of data have different sampling rates, i.e., 120Hz of OptiTrack and around 100Hz of RF-signals, so we re-sample the data to the same sampling rate of 100Hz to correspond RSSI values to each touch point. 2) Feature Extraction: As for a touch gesture, the touch gesture usually starts with the finger hangs over the tag, then the finger presses down to the tag. Considering this continuous process, it is not hard to take the RSSI value just before touching the tag as the reference value for calculating the RSSI deviation, we give an example in Section 7.5. With the reference RSSI value, we can extract the RSSI deviation by subtracting the reference value from the received RSSI values. 3) Curve Fitting: Apply the spline fitting to the derive RSSI deviation pattern, so as to obtain the continuous relationship between the touch point and the RSSI deviation with the discrete RSSI values at finite positions. Considering the uncertain noise during once swipe, a user is required to repeat the swipe across the tag from left to right three times, then we average the three fitting curves to extract the RSSI deviation template. 7.2 Data Preprocessing for Touch Gesture Detection To eliminate the data fluctuation due to the ambient noise in the environment, before performing the fine-grained touch gesture detection, we first use the moving average filter to smooth the data. Specifically, for each sample i, take the average value of the nearest five samples, from i − 2 to i+2, to replace sample i. Meanwhile, similar to the template generation, we determine the reference RSSI value and then calculate the RSSI deviation for the touch position estima￾tion. As we update the reference RSSI value for each touch operation, we can eliminate the major difference of raw RSSI values due to different positions or orientations. Therefore, the derived RSSI deviation mainly reflects the impedance change caused by the finger touch on the tag’s antenna, so it is robust to different positions and orientations. 7.3 Gesture Detection Each touch gesture includes three periods: finger hanging over the tag, finger touching on the tag, finger leaving the tag. Fig. 17 plots the RSSI variations of different touch gestures, beginning with pressing down to the tag and ending with lifting the finger from the tag. Here, the swipe is performed on the half of the tag, i.e., from left to center. It is found that the finger vertical movement, including finger up and finger down, will bring the sudden RSSI variation due to the quick action. Specifically, for click, the RSSI decreases suddenly, then lasts for a short period and recovers to original value suddenly. For press and hold, it is similar to “click”, the only difference is its longer lasting Click / Press and hold Swipe (a) Operation illustration 02468 Time (sec.) -60 -50 -40 -30 RSSI (dBm) Click Press and hold Swipe (b) RSSI variation Fig. 17. Different RSSI variation patterns of three gestures time of the touch situation than “click”. The threshold of the lasting period is set to be 0.7 seconds. While for swipe, the RSSI value decreases suddenly, then changes continuously with different touch positions in several time intervals. If the RSSI changes significantly over successive time intervals, it is identified as the swipe operation. Empirically, the time interval is set to be 0.1 seconds, the number of successive time intervals is 3. Note that, the tag center is a fallible area where the absolute sudden RSSI variation is a bit small, it is easy to mistake the click or press and hold for the noise. According to the yellow swipe data in Fig. 17(b), when the finger leaves the tag from the tag center, the RSSI changes only about 2dB. It is because that when the touch position approaches the tag center, the conjugate mismatching condi￾tion gets better, so the power loss gets less. Especially when the finger just locates at the tag center, touching both the two tag antennas, new conjugate matching will form, and the power loss could be less than 1dB. As the RSSI of a tag in the static environment will change about 1dB due to the ambient noise, a too small RSSI variation threshold will mistake the noise for the touch gesture, so if the range of RSSI values in a short time interval exceeds 2dB, we think the RSSI changes significantly and suddenly, it probably indicates the finger vertical movement or the swipe, instead of the noise. The size of the fallible area is related to the RSSI variation threshold, and it is about 1cm with our settings. Considering the width of the finger, the effective operation size on the used touch tag is around 8.5cm. Thankfully, the size of this fallible area is not large, we suggest to avoid to design a button in this fallible area. Also, as depicted above, the RSSI deviation pattern is symmetric during the swipe across the whole tag, so in our design, we utilize the half part of the tag as the button interface and the other half as the slider interface, an example is illustrated in Fig. 29. This may seem to be the limitation to our work, but in fact, such design is natural and friendly, e.g., the touch bar on Apple MacBook Pro contains both buttons and the slider. How to extend the operation area of the tag and further improve the interaction experience will be studied in our future work. 7.4 Touch Estimation With the known touch interface design, as which part func￾tions as buttons and which part functions as the slider, we can estimate the absolute touch position on the tag by comparing the extracted RSSI deviation with the template. For swipe, by combining the consecutive touch positions, we are able to determine the swipe direction, and calculate the swipe distance as well as the swipe speed, which are the key factors for analyzing the friendly interaction commands