IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 RF-Dial:Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFId Tags Yanling Bu,Student Member,IEEE,Lei Xie,Member,IEEE,Yinyin Gong,Chuyu Wang,Member,IEEE, Lei Yang,Member,IEEE,Jia Liu,Member,IEEE,and Sanglu Lu,Member,IEEE Abstract-With the rising of demands for novel human-computer interaction approaches in the 2D plane,a number of intelligent devices come into being.For example,Microsoft Surface Dial supports simple clicks and rotations for the interaction with computer. However,these approaches are dedicated devices,and they might require batteries or have limited functions.In this paper,we propose RF-Dial to realize a light-weight,battery-free and functional 2D human-computer interaction solution via commercial off-the-shelf (COTS)passive RFID tags.What RF-Dial shines is that it can easily turn an ordinary object,e.g.,a board eraser,into an intelligent interaction device.By deploying a tag array on the side face of the object together with a dipole tag on the top face,RF-Dial cannot only track the rigid motion of the object but also detect the touch gesture of a user on the surface of the object,including translation, rotation,click,press and hold,and swipe.To do the motion tracking,RF-Dial builds a phase-based model that captures the translation and the rotation of the tagged object simultaneously,by jointly exploiting the information of phase variations and the topology of the tag array.To detect the touch gesture,RF-Dial builds an RSSI-based model that uses the impact of the touching finger on the tag antenna's impedance to estimate the touch position in real time,which is robust to environmental factors like position or orientation.We implemented a prototype of RF-Dial with commodity RFID devices.Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking,with a small error of 0.6cm for the translation tracking,and a small error of 1.9 degrees for the rotation estimation. Besides,RF-Dial can also detect the touch gesture accurately,as the 90 percent of touch position errors are less than 2.09mm. Index Terms-RFID,human-computer interaction,tag array,translation,rotation,coupling effect,touch gesture. ◆ INTRODUCTION TN modern times,the widely used approaches for the novel HCI designs [2-18].RFID can even work in the non- human-computer interaction (HCI)are operated in the line-of-sight situation due to its backscatter communication. 2D plane,like the touch screen and the mouse.By moving Therefore,we hope to use RFID to answer such a question: or stroking these interaction devices,users can access the "Is it possible to design a battery-free and light-weight solution objects in the computer and manipulate them conveniently.to the 2D human-computer interaction,thereby even an ordinary With the rise of the computer aided art design and other object can be easily turned into an intelligent interaction device?" novel applications,a number of intelligent devices have In this paper,we propose RF-Dial to realize a novel come into being as the response to the demand for brand-2D human-computer interaction solution via COTS passive new 2D interaction solutions.For example,Microsoft Sur- RFID tags.We attach a tag array to the side face together face Dial [1]emerged in 2016,supporting simple clicks and with one tag on the top face of an object,denote them as rotations for the natural and friendly interaction. movement tags and the touch tag,respectively.As shown The latest HCI approaches are mainly based on the in Fig.1,we deploy two RFID antennas orthogonally to computer vision or sensors.For computer vision-based realize our vision.Specifically,we continuously track the approaches,they use cameras to monitor the movement rigid motion of the tagged object with movement tags,in- of limbs or fingers.However,they are mainly limited by cluding the translation and the rotation simultaneously,and privacy concerns,the light condition and the viewing an- detect the touch gesture with the touch tag,including the gle.For sensor-based approaches,they use commercial- click,the press and hold,and the swipe.In this way,an off-the-shelf (COTS)sensors like inertial sensors to track ordinary object such as a candy box can be turned into an the movement of devices.Their main constraints are the intelligent interaction device.For example,we can realize limited battery life and the high hardware cost.Thankfully, an functional drawing application with RF-Dial.It tracks RFID provides the battery-free sensing technology to enable the translation of movement tags to draw lines,and adjusts the line color automatically by the rotation just during the drawing process;The touch tag functions as buttons and Yanling Bu,Lei Xie,Yinyin Gong,Chuyu Wang,Jia Liu,and Sanglu Lu are with the State Key Laboratory for Novel Softiare Technology,Nanjing sliders,receiving the user's commands,i.e.,adjusting the University,China. line color and width,as shown in the case study (Sec- E-mail: yanling@smail.nju.edu.cn, Ixie@nju.edu.cn, yy-tion 9.4).Technically,based on RF-signals from movement gong@dislab.nju.edu.cn, chuyu@nju.edu.cn, jialiu@nju.edu.cn, tags,we build a rigid transformation model to reflect the sanglu@nju.edu.cn. Lei Yang is with the Department of Computing,The Hong Kong Poly- relationship between the motion of the tagged object and technic University,Hong Kong,China. the corresponding phase variations of each movement tag in E-mail:young @tagsys.org. the tag array.As the movement tags form a tag array with .Lei Xie is the corresponding author. the fixed topology,we can derive the translation and the
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 1 RF-Dial: Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFID Tags Yanling Bu, Student Member, IEEE, Lei Xie, Member, IEEE, Yinyin Gong, Chuyu Wang, Member, IEEE, Lei Yang, Member, IEEE, Jia Liu, Member, IEEE, and Sanglu Lu, Member, IEEE Abstract—With the rising of demands for novel human-computer interaction approaches in the 2D plane, a number of intelligent devices come into being. For example, Microsoft Surface Dial supports simple clicks and rotations for the interaction with computer. However, these approaches are dedicated devices, and they might require batteries or have limited functions. In this paper, we propose RF-Dial to realize a light-weight, battery-free and functional 2D human-computer interaction solution via commercial off-the-shelf (COTS) passive RFID tags. What RF-Dial shines is that it can easily turn an ordinary object, e.g., a board eraser, into an intelligent interaction device. By deploying a tag array on the side face of the object together with a dipole tag on the top face, RF-Dial cannot only track the rigid motion of the object but also detect the touch gesture of a user on the surface of the object, including translation, rotation, click, press and hold, and swipe. To do the motion tracking, RF-Dial builds a phase-based model that captures the translation and the rotation of the tagged object simultaneously, by jointly exploiting the information of phase variations and the topology of the tag array. To detect the touch gesture, RF-Dial builds an RSSI-based model that uses the impact of the touching finger on the tag antenna’s impedance to estimate the touch position in real time, which is robust to environmental factors like position or orientation. We implemented a prototype of RF-Dial with commodity RFID devices. Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking, with a small error of 0.6cm for the translation tracking, and a small error of 1.9 degrees for the rotation estimation. Besides, RF-Dial can also detect the touch gesture accurately, as the 90 percent of touch position errors are less than 2.09mm. Index Terms—RFID, human-computer interaction, tag array, translation, rotation, coupling effect, touch gesture. ✦ 1 INTRODUCTION I N modern times, the widely used approaches for the human-computer interaction (HCI) are operated in the 2D plane, like the touch screen and the mouse. By moving or stroking these interaction devices, users can access the objects in the computer and manipulate them conveniently. With the rise of the computer aided art design and other novel applications, a number of intelligent devices have come into being as the response to the demand for brandnew 2D interaction solutions. For example, Microsoft Surface Dial [1] emerged in 2016, supporting simple clicks and rotations for the natural and friendly interaction. The latest HCI approaches are mainly based on the computer vision or sensors. For computer vision-based approaches, they use cameras to monitor the movement of limbs or fingers. However, they are mainly limited by privacy concerns, the light condition and the viewing angle. For sensor-based approaches, they use commercialoff-the-shelf (COTS) sensors like inertial sensors to track the movement of devices. Their main constraints are the limited battery life and the high hardware cost. Thankfully, RFID provides the battery-free sensing technology to enable • Yanling Bu, Lei Xie, Yinyin Gong, Chuyu Wang, Jia Liu, and Sanglu Lu are with the State Key Laboratory for Novel Software Technology, Nanjing University, China. E-mail: yanling@smail.nju.edu.cn, lxie@nju.edu.cn, yygong@dislab.nju.edu.cn, chuyu@nju.edu.cn, jialiu@nju.edu.cn, sanglu@nju.edu.cn. • Lei Yang is with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China. E-mail: young@tagsys.org. • Lei Xie is the corresponding author. novel HCI designs [2–18]. RFID can even work in the nonline-of-sight situation due to its backscatter communication. Therefore, we hope to use RFID to answer such a question: “Is it possible to design a battery-free and light-weight solution to the 2D human-computer interaction, thereby even an ordinary object can be easily turned into an intelligent interaction device?”. In this paper, we propose RF-Dial to realize a novel 2D human-computer interaction solution via COTS passive RFID tags. We attach a tag array to the side face together with one tag on the top face of an object, denote them as movement tags and the touch tag, respectively. As shown in Fig. 1, we deploy two RFID antennas orthogonally to realize our vision. Specifically, we continuously track the rigid motion of the tagged object with movement tags, including the translation and the rotation simultaneously, and detect the touch gesture with the touch tag, including the click, the press and hold, and the swipe. In this way, an ordinary object such as a candy box can be turned into an intelligent interaction device. For example, we can realize an functional drawing application with RF-Dial. It tracks the translation of movement tags to draw lines, and adjusts the line color automatically by the rotation just during the drawing process; The touch tag functions as buttons and sliders, receiving the user’s commands, i.e., adjusting the line color and width, as shown in the case study (Section 9.4). Technically, based on RF-signals from movement tags, we build a rigid transformation model to reflect the relationship between the motion of the tagged object and the corresponding phase variations of each movement tag in the tag array. As the movement tags form a tag array with the fixed topology, we can derive the translation and the
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 Tagged Daily absolute touch position,we observe that the RSSI deviation during the swipe is position-independent and orientation- insensitive.We explore the signal variation when touching different positions on the tag,and build an RSSI-based MeTg Touch Ges model to verify the robustness of the RSSI deviation,which Track Rigid Touch Tag: Modon is mainly related to impedance change due to touch.Hence, Deteer Toueh based on the RSSI deviation,we can identify the absolute Gesture candidate touch positions.Note that,as the used linear Fig.1.An example application scenario of RF-Dial tag has the dipole antenna,the RSSI variation during the swipe across the tag forms a symmetric -wave pattern.To rotation of the tagged object for each snapshot based on the eliminate such ambiguity of touch positions,we use half of rigid transformation model.Note that,the phase contours the tag as buttons and the other half as the slider.Referring of RF-signals vary at different positions in the scanning to the RSSI variation,it is easy to determine which gesture area,the relationship between the tag movement and the is performed,thereby we can identify which part of the tag phase variation is different,regarding to which we split is touched and further derive the unique touch position.By the effective scanning area into linear region and non-linear tracking the consecutive touch positions of the swipe,we region.Meanwhile,based on RF-signals from the touch tag, are able to estimate the swipe direction and distance. we build an RSSI-based model to depict the relationship be- Overall,we make the following three main contribu- tween the touch position on the tag and the corresponding tions.First,we propose a novel interaction scheme via RFID RSSI variation of received RF-signals.According to the RSSI- technology,supporting the rigid motion tracking and the based model,the RSSI deviation is mainly related to the touch gesture detection.An ordinary object can be turned impedance change when touching the tag,so it is position- into an intelligent HCI device via attaching a tag array independent and orientation-insensitive.Consequently,we on the side face together with one linear tag on the top can rely on only one general RSSI deviation template to ac- face,denoted as movement tags and the touch tag,respec- curately and robustly determine the touch position,without tively.Second,we build a phase-based model to reflect the the known start touch position or the fixed tag deployment, relationship between the motion of tagged object and the among the whole monitoring area. corresponding phase variations of movement tags in the There are three key challenges to realize RF-Dial.1) array.We also build an RSSI-based model to depict the rela- How to estimate the rigid motion of the tagged object based tionship between the touch position and the corresponding on RE-signals of tags,including the translation and rotation RSSI deviation of the touch tag.Third,we implemented a simultaneously,is a key problem.To tackle this challenge,we prototype system of RF-Dial and evaluated its performance build a rigid transformation model to reflect the relationship in the real environment.Extensive experiments show that between the motion of the tagged object and the correspond- RF-Dial achieves an accurate rigid motion tracking,with a ing phase variations of each movement tag in the tag array. small error of 0.6cm for translation tracking,and a small As the topology of the movement tag array is fixed,we are error of 1.9 degrees for rotation estimation.Besides,RF-Dial able to decompose the rigid motion of the tagged object can also detect the touch gesture accurately,as the 90 percent referring to the phase variations of at least two movement of touch position errors are less than 2.09mm. tags,and then derive the translation and the rotation of the tagged object for each snapshot during the motion.2)How 2 RELATED WORK to address the variation of phase contours at different positions RFID-based Localization:A straightforward solution for in the effective scanning area is a key problem.Our empirical RFID-based human-computer interaction is to utilize RFID study shows that the phase contours are close to concentric localization schemes to directly locate tagged objects [2- circles with the antenna at the center.Hence,even for the 6,20-26].State-of-the-art systems mainly use phase values same rigid motion of the tagged object,the antenna could for the accurate localization [2-4,6,22,23].PinIt [2]uses collect different phase variations at different positions.To multi-path profiles of tags to accurately locate tags with the tackle this challenge,regarding to the relationship between synthetic aperture radar created via the antenna motion the tag movement and the phase variation,we split the Rather than the absolute localization,STPP [6]is the first whole scanning area into linear region and non-linear region. work to tackle 2D relative localization,which uses the Specifically,the tag movement in the linear region is linear spatial-temporal dynamics in the phase profiles to identify to the phase variation,thus we can extract the tag movement the relative positions of tags.More than only using the based on the phase variations detected from the two orthog- phase information,RFind [20]leverages the complete phys- onal antennas.While in the non-linear region,we locate the ical properties of RF-signals to realize the ultra-wideband tag first,then extract the tag movement based on the phase localization.RFind is capable of emulating over 220MHz of contours at the tag's position.3)How to obtain the absolute bandwidth without changing the tag and remains compliant touch position of the tag when the tag moves to any position with current regulations.However,most approaches figure with different orientations within the monitoring area is a key out the absolute positions of tags in a separate manner, problem.Existing work like [19]leverages the phase variation whereas RF-Dial aims to track the movement of the tag to detect the touch gesture,however,the phase is sensitive array in a comprehensive manner.By referring to the fixed to the position and orientation of the tag,so it can only topology of tag array,RF-Dial can accurately track the rigid track the touch position with the known start touch point transformation of tagged object,including the translation of a fixed tag.To tackle the challenge of determining the and rotation simultaneously
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 2 RFID Antenna Pair Track Rigid Motion · Rotation · Translation Rigid Motion Touch Tag: Detect Touch Gesture Movement Tags: Tagged Daily Objects Touch Gesture · Click · Press & Hold · Swipe Fig. 1. An example application scenario of RF-Dial rotation of the tagged object for each snapshot based on the rigid transformation model. Note that, the phase contours of RF-signals vary at different positions in the scanning area, the relationship between the tag movement and the phase variation is different, regarding to which we split the effective scanning area into linear region and non-linear region. Meanwhile, based on RF-signals from the touch tag, we build an RSSI-based model to depict the relationship between the touch position on the tag and the corresponding RSSI variation of received RF-signals. According to the RSSIbased model, the RSSI deviation is mainly related to the impedance change when touching the tag, so it is positionindependent and orientation-insensitive. Consequently, we can rely on only one general RSSI deviation template to accurately and robustly determine the touch position, without the known start touch position or the fixed tag deployment, among the whole monitoring area. There are three key challenges to realize RF-Dial. 1) How to estimate the rigid motion of the tagged object based on RF-signals of tags, including the translation and rotation simultaneously, is a key problem. To tackle this challenge, we build a rigid transformation model to reflect the relationship between the motion of the tagged object and the corresponding phase variations of each movement tag in the tag array. As the topology of the movement tag array is fixed, we are able to decompose the rigid motion of the tagged object referring to the phase variations of at least two movement tags, and then derive the translation and the rotation of the tagged object for each snapshot during the motion. 2) How to address the variation of phase contours at different positions in the effective scanning area is a key problem. Our empirical study shows that the phase contours are close to concentric circles with the antenna at the center. Hence, even for the same rigid motion of the tagged object, the antenna could collect different phase variations at different positions. To tackle this challenge, regarding to the relationship between the tag movement and the phase variation, we split the whole scanning area into linear region and non-linear region. Specifically, the tag movement in the linear region is linear to the phase variation, thus we can extract the tag movement based on the phase variations detected from the two orthogonal antennas. While in the non-linear region, we locate the tag first, then extract the tag movement based on the phase contours at the tag’s position. 3) How to obtain the absolute touch position of the tag when the tag moves to any position with different orientations within the monitoring area is a key problem. Existing work like [19] leverages the phase variation to detect the touch gesture, however, the phase is sensitive to the position and orientation of the tag, so it can only track the touch position with the known start touch point of a fixed tag. To tackle the challenge of determining the absolute touch position, we observe that the RSSI deviation during the swipe is position-independent and orientationinsensitive. We explore the signal variation when touching different positions on the tag, and build an RSSI-based model to verify the robustness of the RSSI deviation, which is mainly related to impedance change due to touch. Hence, based on the RSSI deviation, we can identify the absolute candidate touch positions. Note that, as the used linear tag has the dipole antenna, the RSSI variation during the swipe across the tag forms a symmetric Ω-wave pattern. To eliminate such ambiguity of touch positions, we use half of the tag as buttons and the other half as the slider. Referring to the RSSI variation, it is easy to determine which gesture is performed, thereby we can identify which part of the tag is touched and further derive the unique touch position. By tracking the consecutive touch positions of the swipe, we are able to estimate the swipe direction and distance. Overall, we make the following three main contributions. First, we propose a novel interaction scheme via RFID technology, supporting the rigid motion tracking and the touch gesture detection. An ordinary object can be turned into an intelligent HCI device via attaching a tag array on the side face together with one linear tag on the top face, denoted as movement tags and the touch tag, respectively. Second, we build a phase-based model to reflect the relationship between the motion of tagged object and the corresponding phase variations of movement tags in the array. We also build an RSSI-based model to depict the relationship between the touch position and the corresponding RSSI deviation of the touch tag. Third, we implemented a prototype system of RF-Dial and evaluated its performance in the real environment. Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking, with a small error of 0.6cm for translation tracking, and a small error of 1.9 degrees for rotation estimation. Besides, RF-Dial can also detect the touch gesture accurately, as the 90 percent of touch position errors are less than 2.09mm. 2 RELATED WORK RFID-based Localization: A straightforward solution for RFID-based human-computer interaction is to utilize RFID localization schemes to directly locate tagged objects [2– 6, 20–26]. State-of-the-art systems mainly use phase values for the accurate localization [2–4, 6, 22, 23]. PinIt [2] uses multi-path profiles of tags to accurately locate tags with the synthetic aperture radar created via the antenna motion. Rather than the absolute localization, STPP [6] is the first work to tackle 2D relative localization, which uses the spatial-temporal dynamics in the phase profiles to identify the relative positions of tags. More than only using the phase information, RFind [20] leverages the complete physical properties of RF-signals to realize the ultra-wideband localization. RFind is capable of emulating over 220MHz of bandwidth without changing the tag and remains compliant with current regulations. However, most approaches figure out the absolute positions of tags in a separate manner, whereas RF-Dial aims to track the movement of the tag array in a comprehensive manner. By referring to the fixed topology of tag array, RF-Dial can accurately track the rigid transformation of tagged object, including the translation and rotation simultaneously
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 RFID-based Motion Tracking:Prior RFID-based motion RF-signals Rigid Motion Tracking tracking systems propose various approaches for the trajec- tory tracking [7-12,27]and orientation tracking [13,14,28]. Data Preprocessing Motion or Motion Representative work like RF-IDraw [7]and PolarDraw [8] Data Segmentation touch? use a single tag to reconstruct the handwriting,regarding Template Generation Initial State Estimation the tag as a mass point for the motion tracking.Tag- Re-sampling Compass [14]uses a single tag to determine the orientation Feature Extraction Movement Tracking and position of the tagged object based on polarization Tag Movement Derivation properties of RF waves.Further,recent work uses the tag Fitting吗 Rigid Transformation array to track the trajectory or orientation of the mov- Touch Gesture Detection Tracking ing object.Specifically,Pantomime [9]enables the accurate Datu Preprocessing trajectory tracking of the tagged object with a tag array, Movement Calibration using a multiple tag single antenna system.Tagyro [13] Gesture Detection Outlier Detection realizes the 3D orientation tracking with an array of RFID tags,by converting the real-time phase offsets between tags Touch Estimation Outlier Elimination into the orientation angle.However,these approaches track either the trajectory or the orientation of the moving object, Interaction Commands without detecting the translation and rotation of the tagged Click Press and hold Swipe Translation S Rotation R object simultaneously.Tagball 16 is the closest work to RF-Dial,which studies the motion behavior,including the Fig.2.System framework translation and rotation,of a ball attached with a tag array. TABLE 1 However,Tagball solves the problem by the absolute local- Signal variation of different operations ization on multiple tags.Specifically,it first estimates the Operations Considerable Signal Variation absolute positions of multiple tags via the phase values,and Movement Tag Touch Tag None No No then figures out the translation and rotation of the tagged Movement Yes Yes object based on the estimated positions of tags.Hence,the Touch No Yes localization errors are further introduced to the estimation of the translation and rotation.Therefore,it requires plenty insensitive.With only one general RSSI deviation template, of tags,i.e.,12 tags in total,to provide enough data to the we can accomplish the light-weight and fine-grained touch Extended Kalman Filter-based tracking model to guarantee detection accurately among the whole operating area. the tracking accuracy.In comparison,RF-Dial tracks the translation and rotation of tagged objects simultaneously by 3 SYSTEM OVERVIEW directly referring to phase variations from at least two tags, thus it achieves more accuracy in the motion tracking. RF-Dial is designed to provide two functions:one is to track RFID-based Touch Sensing:Besides the traditional the rigid motion of the tagged object,including the transla- studies on localization or tracking,recently researchers also tion and the rotation,the other is to detect the touch gesture have studied the tag's physical change when the conductor on one linear tag,such as the click,the press and hold,and touches the tag,i.e.,the liquid and human beings,and tried the swipe.The basic idea is to use a tag array attached on to utilize such characteristics for applications like liquid the side face of an object to track its movement,and use detection [29]and touch interaction [19,30,31].PaperID [30] another single tag attached on the top face of the object to provides the capability to use COTS tags to sense the finger detect the touch gesture,as shown in Fig.1.Suppose the touch,swipe touch,and other gestures by the support tags in a array that track the motion are movement fags,the vector machine(SVM).But this sensing capability is very single tag that detects the touch gesture is the touch tag.To coarse,i.e.,it only can detect whether the touch happens reduce the mutual coupling between tags,we separate the but cannot determine where the exact touch position is on touch tag and the movement tags with enough distances, the tag.Meanwhile,the machine learning algorithm requires i.e.,more than 6cm of height difference.Fig.2 illustrates the high training overhead.RIO [19]observes that when a the system framework.After receiving the raw RF-signals human finger touches the tag,the tag's impedance changes, from tags,we first determine which kind of the operation, which causes the phase change among received RF-signals i.e.,touch or movement.The intuition is that for the touch correspondingly.Based on the phase variation template,it gesture,the signal of the touch tag changes significantly tracks the finger position during a swipe using the segmen- but the signals of movement tags keep steady,while for the tal dynamic time warping (SDTW)method.However,RIO motion,the signal of either the touch tag or the movement can only work with the known start finger position,other- tags changes along with the continuous motion,as shown in wise it cannot use the SDTW to track the finger position. TABLE 1.Therefore,by extracting the features of the signal Also,the phase is sensitive to the distance from the tag and variation of tags,such as the phase or RSSI difference in antenna,so the template is required to be updated when time intervals,we can determine which operation the user the tag is moved to different positions.Whereas,RF-Dial is performs using the random forest algorithm.Details are designed to track the absolute finger position without the shown in Section 7.5. known start position.Our solution is based on the RSSI de- As shown in Fig.2,there are two main function modules viation resulted from the impedance change when the touch to realize our goal of the rigid motion tracking and the touch happens,which is position-independent and orientation- gesture detection separately.In the following,we first build
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 3 RFID-based Motion Tracking: Prior RFID-based motion tracking systems propose various approaches for the trajectory tracking [7–12, 27] and orientation tracking [13, 14, 28]. Representative work like RF-IDraw [7] and PolarDraw [8] use a single tag to reconstruct the handwriting, regarding the tag as a mass point for the motion tracking. TagCompass [14] uses a single tag to determine the orientation and position of the tagged object based on polarization properties of RF waves. Further, recent work uses the tag array to track the trajectory or orientation of the moving object. Specifically, Pantomime [9] enables the accurate trajectory tracking of the tagged object with a tag array, using a multiple tag single antenna system. Tagyro [13] realizes the 3D orientation tracking with an array of RFID tags, by converting the real-time phase offsets between tags into the orientation angle. However, these approaches track either the trajectory or the orientation of the moving object, without detecting the translation and rotation of the tagged object simultaneously. Tagball [16] is the closest work to RF-Dial, which studies the motion behavior, including the translation and rotation, of a ball attached with a tag array. However, Tagball solves the problem by the absolute localization on multiple tags. Specifically, it first estimates the absolute positions of multiple tags via the phase values, and then figures out the translation and rotation of the tagged object based on the estimated positions of tags. Hence, the localization errors are further introduced to the estimation of the translation and rotation. Therefore, it requires plenty of tags, i.e., 12 tags in total, to provide enough data to the Extended Kalman Filter-based tracking model to guarantee the tracking accuracy. In comparison, RF-Dial tracks the translation and rotation of tagged objects simultaneously by directly referring to phase variations from at least two tags, thus it achieves more accuracy in the motion tracking. RFID-based Touch Sensing: Besides the traditional studies on localization or tracking, recently researchers also have studied the tag’s physical change when the conductor touches the tag, i.e., the liquid and human beings, and tried to utilize such characteristics for applications like liquid detection [29] and touch interaction [19, 30, 31]. PaperID [30] provides the capability to use COTS tags to sense the finger touch, swipe touch, and other gestures by the support vector machine (SVM). But this sensing capability is very coarse, i.e., it only can detect whether the touch happens but cannot determine where the exact touch position is on the tag. Meanwhile, the machine learning algorithm requires the high training overhead. RIO [19] observes that when a human finger touches the tag, the tag’s impedance changes, which causes the phase change among received RF-signals correspondingly. Based on the phase variation template, it tracks the finger position during a swipe using the segmental dynamic time warping (SDTW) method. However, RIO can only work with the known start finger position, otherwise it cannot use the SDTW to track the finger position. Also, the phase is sensitive to the distance from the tag and antenna, so the template is required to be updated when the tag is moved to different positions. Whereas, RF-Dial is designed to track the absolute finger position without the known start position. Our solution is based on the RSSI deviation resulted from the impedance change when the touch happens, which is position-independent and orientationData Preprocessing Movement Tracking Movement Calibration Outlier Detection Outlier Elimination Tag Movement Derivation Rigid Transformation Tracking Initial State Estimation Data Segmentation Rigid Motion Tracking Touch Gesture Detection Data Preprocessing Gesture Detection Touch Estimation Motion or touch? Motion Touch RF-signals Interaction Commands Click Press and hold Swipe Translation �� Rotation �� Feature Extraction Re-sampling Fitting Template Generation Fig. 2. System framework TABLE 1 Signal variation of different operations Operations Considerable Signal Variation Movement Tag Touch Tag None No No Movement Yes Yes Touch No Yes insensitive. With only one general RSSI deviation template, we can accomplish the light-weight and fine-grained touch detection accurately among the whole operating area. 3 SYSTEM OVERVIEW RF-Dial is designed to provide two functions: one is to track the rigid motion of the tagged object, including the translation and the rotation, the other is to detect the touch gesture on one linear tag, such as the click, the press and hold, and the swipe. The basic idea is to use a tag array attached on the side face of an object to track its movement, and use another single tag attached on the top face of the object to detect the touch gesture, as shown in Fig. 1. Suppose the tags in a array that track the motion are movement tags, the single tag that detects the touch gesture is the touch tag. To reduce the mutual coupling between tags, we separate the touch tag and the movement tags with enough distances, i.e., more than 6cm of height difference. Fig. 2 illustrates the system framework. After receiving the raw RF-signals from tags, we first determine which kind of the operation, i.e., touch or movement. The intuition is that for the touch gesture, the signal of the touch tag changes significantly but the signals of movement tags keep steady, while for the motion, the signal of either the touch tag or the movement tags changes along with the continuous motion, as shown in TABLE 1. Therefore, by extracting the features of the signal variation of tags, such as the phase or RSSI difference in time intervals, we can determine which operation the user performs using the random forest algorithm. Details are shown in Section 7.5. As shown in Fig. 2, there are two main function modules to realize our goal of the rigid motion tracking and the touch gesture detection separately. In the following, we first build
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 a phase-based model for the rigid motion tracking in Sec- Central Beam Region tion 4,and propose a tracking solution based on the motion 180H model in Section 5.Then,we build an RSSI-based model 160 for the touch gesture detection in Section 6,and propose the corresponding solution to realize the absolute touch position detection robustly and accurately in Section 7. 120 4 RIGID MOTION TRACKING 1005 -100-80 -60 -40 -20 0 20 100 In this section,we propose the definition of linear region and Fig.3.Phase contours of RF-signals non-linear region based on phase contour variations.Then, we illustrate the relationship between the tag movement and phase variation in different regions.Further,we model the rigid motion to decompose the translation and rotation. Linear Region 0.6x0.6m2 4.1 Linear Region and Non-linear Region In RFID systems,the RF phase is a common attribute of the Non-linear Region wireless signal,ranging from 0 to 2m.It is very sensitive to the tag-antenna distance.Suppose the distance between the tag and the antenna is d,so the signal traverses a distance Fig.4.Effective scanning area:linear region vs non-linear region of 2d in the backscatter communication.Then,the phase provided by the antenna can be expressed as: of antenna Ay smaller than a certain value,we think this position belongs to the central beam of Ay.Also,as shown 0= ×2d+4 mod 2. (1) in Fig.3,the width of the central beam changes with the perpendicular distance.The larger the perpendicular where A is the wavelength,u represents the phase offset distance is,the wider the central beam is at that distance,the caused by the diversity of hardware characteristics.Accord- central beam region is trapezoidal.Therefore,taking Fig.4, ing to the phase expression in Eq.(1),besides the diversity if setting the small displacement as 5cm,the phase different term,the phase value mainly depends on the distance threshold as 0.4 radians,for the position with the distance between the tag and the antenna.Therefore,the phase of 1.2m from two antennas,the size of the linear region is contours should form concentric circles with the antenna about 0.6 x 0.6m2,centered at that position.Apart from the at the center in an ideal situation.We thus conduct an linear region,the phase variations in the other scanning area experiment to validate the above hypothesis.We build a 2D are not linear to the displacements along either the X-axis coordinate system according to the parallel direction (X- or Y-axis,they depend on the exact tag position instead. axis)and perpendicular direction (Y-axis)of the antenna, Hence,we denote the other area as the non-linear region. and set the origin (0,0)at the center of the antenna.Then, we collect phase values in a rectangle space in front of the 4.2 Rigid Transformation antenna,ranging from -100cm to 100cm along the X-axis During the continuous movement of an object,its position and from 100cm to 180cm along the Y-axis,the step is 5cm. and orientation are changing all the time.For a rigid body, The collected phase values are plotted in Fig.3.Based on such change of the position and orientation in the 2D the experiment results,we have the following observation: space can be defined by the rigid transformation R,S, Observation 1:The phase contours are very close to concen- where R is a 2 x 2 rotation matrix and S is a 2 x 1 tric circles with the antenna at the center.Besides,in the central translation matrix.Here,the rotation means a circular move- beam region marked with blue lines in Fig.3,the phase contours ment that the device rotates around a rotation center,and are almost parallel to each other and stretching along the X-axis. the translation means a linear movement that every point That is,in this region,the phase can be regarded as linearly related of the device moves with the same displacement.As the to the perpendicular distance from the tag to the antenna plane. continuous movement of an object consists of a series of As shown in Fig.4,assume that two antennas Ar and instant movements at different time,we denote the instant Ay are deployed in a mutually orthogonal manner and movement as the micro-movement,each micro-movement can separated with a fairly large distance.Then,according to be expressed with the rotation and translation.Thus,we can Observation 1,in the intersection area of the central beams use the rigid transformation to depict the micro-movement. of the two antennas,the displacement of a tag along the By attaching a tag array on an object,it is possible to X-axis and Y-axis should be linear to the phase variations track the rigid transformation based on the movement of received by the antenna Ar and Au,respectively.We thus each tag in the array.Note that,different from the rigid body, denote this intersection region as the linear region.The size i.e.,the tagged object,the tag attached on the object actually of the linear region depends on two factors:the central beam represents a single point of the object,so its movement can region of each antenna,and the perpendicular distance to be regarded as the particle movement,which only has the each antenna.Specifically,the central beam region relies on translation rather than the rotation.E.g.,assume an object the tolerance of the small displacement along the horizontal is attached with a tag array with the layout of rectangle, direction for the antenna.E.g.,if the small displacement as shown in Fig.5,the tags are denoted as solid points on along the X-axis at one position incurs the phase difference the rectangle.For any micro-movement in the continuous
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 4 a phase-based model for the rigid motion tracking in Section 4, and propose a tracking solution based on the motion model in Section 5. Then, we build an RSSI-based model for the touch gesture detection in Section 6, and propose the corresponding solution to realize the absolute touch position detection robustly and accurately in Section 7. 4 RIGID MOTION TRACKING In this section, we propose the definition of linear region and non-linear region based on phase contour variations. Then, we illustrate the relationship between the tag movement and phase variation in different regions. Further, we model the rigid motion to decompose the translation and rotation. 4.1 Linear Region and Non-linear Region In RFID systems, the RF phase is a common attribute of the wireless signal, ranging from 0 to 2π. It is very sensitive to the tag-antenna distance. Suppose the distance between the tag and the antenna is d, so the signal traverses a distance of 2d in the backscatter communication. Then, the phase provided by the antenna can be expressed as: θ = 2π λ × 2d + µ mod 2π, (1) where λ is the wavelength, µ represents the phase offset caused by the diversity of hardware characteristics. According to the phase expression in Eq. (1), besides the diversity term, the phase value mainly depends on the distance between the tag and the antenna. Therefore, the phase contours should form concentric circles with the antenna at the center in an ideal situation. We thus conduct an experiment to validate the above hypothesis. We build a 2D coordinate system according to the parallel direction (Xaxis) and perpendicular direction (Y -axis) of the antenna, and set the origin (0, 0) at the center of the antenna. Then, we collect phase values in a rectangle space in front of the antenna, ranging from −100cm to 100cm along the X-axis and from 100cm to 180cm along the Y -axis, the step is 5cm. The collected phase values are plotted in Fig. 3. Based on the experiment results, we have the following observation: Observation 1: The phase contours are very close to concentric circles with the antenna at the center. Besides, in the central beam region marked with blue lines in Fig. 3, the phase contours are almost parallel to each other and stretching along the X-axis. That is, in this region, the phase can be regarded as linearly related to the perpendicular distance from the tag to the antenna plane. As shown in Fig. 4, assume that two antennas Ax and Ay are deployed in a mutually orthogonal manner and separated with a fairly large distance. Then, according to Observation 1, in the intersection area of the central beams of the two antennas, the displacement of a tag along the X-axis and Y -axis should be linear to the phase variations received by the antenna Ax and Ay, respectively. We thus denote this intersection region as the linear region. The size of the linear region depends on two factors: the central beam region of each antenna, and the perpendicular distance to each antenna. Specifically, the central beam region relies on the tolerance of the small displacement along the horizontal direction for the antenna. E.g., if the small displacement along the X-axis at one position incurs the phase difference -100 -80 -60 -40 -20 0 20 40 60 80 100 100 120 140 160 180 1 2 3 4 5 6 Central Beam Region Fig. 3. Phase contours of RF-signals � � �! Linear Region 1.2� 1.2� 0.6×0.6�! Non-linear Region �" � Fig. 4. Effective scanning area: linear region vs non-linear region of antenna Ay smaller than a certain value, we think this position belongs to the central beam of Ay. Also, as shown in Fig. 3, the width of the central beam changes with the perpendicular distance. The larger the perpendicular distance is, the wider the central beam is at that distance, the central beam region is trapezoidal. Therefore, taking Fig. 4, if setting the small displacement as 5cm, the phase different threshold as 0.4 radians, for the position with the distance of 1.2m from two antennas, the size of the linear region is about 0.6 × 0.6m2 , centered at that position. Apart from the linear region, the phase variations in the other scanning area are not linear to the displacements along either the X-axis or Y -axis, they depend on the exact tag position instead. Hence, we denote the other area as the non-linear region. 4.2 Rigid Transformation During the continuous movement of an object, its position and orientation are changing all the time. For a rigid body, such change of the position and orientation in the 2D space can be defined by the rigid transformation R, S , where R is a 2 × 2 rotation matrix and S is a 2 × 1 translation matrix. Here, the rotation means a circular movement that the device rotates around a rotation center, and the translation means a linear movement that every point of the device moves with the same displacement. As the continuous movement of an object consists of a series of instant movements at different time, we denote the instant movement as the micro-movement, each micro-movement can be expressed with the rotation and translation. Thus, we can use the rigid transformation to depict the micro-movement. By attaching a tag array on an object, it is possible to track the rigid transformation based on the movement of each tag in the array. Note that, different from the rigid body, i.e., the tagged object, the tag attached on the object actually represents a single point of the object, so its movement can be regarded as the particle movement, which only has the translation rather than the rotation. E.g., assume an object is attached with a tag array with the layout of rectangle, as shown in Fig. 5, the tags are denoted as solid points on the rectangle. For any micro-movement in the continuous
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 5 4.3.2 Tag Movement in the Non-linear Region In the non-linear region,since the corresponding phase vari- ations are not linear to the tag movement,we need to figure out their relationship according to the geometric property. Given the phase variations Ar and Aby respectively col- Trajectory lected from the two orthogonally deployed antennas A.and 4 X Ay,according to Eq.(4),we have: Fig.5.Rigid transformation in con-Fig.6.Relationship between tag tinuous moving process movement and phase variation T1△x十△y= 入0r 4π (6 movement,it can be intuitively observed that,the rigid 入9y x,△x+,△y=4 transformation of the tagged object,including the translation and rotation,can be derived from the movement of different tags. where(zlz,.〉and〈rl,h,）denote the normalized vec- tor for the polar axis AP from the antenna A and Ay, 4.3 Model of Tag Movement and Phase Variation respectively.Therefore,as long as the starting position of According to Observation 1,the phase contours can be de- movement s,i.e,P,is known,the values of (,y)and picted as concentric circles with the antenna at the center. can be figured out.Then,by solving the linear Thus,we can build a polar coordinate system by setting equations in Eq.(6),we can directly compute [Ar,Ay]T. the center of the antenna as the origin.Then,given a tag movement s,we can further depict the relationship between 4.4 Model of Rigid Motion Decomposition the phase variation and the movement s in this polar coordi- As aforementioned,during the continuous moving process nate system.As shown in Fig.6,the antenna is deployed at of the rigid body,the micro-movement can be defined by position A,we use the vector s to denote the tag movement, the rigid transformation including the rotation and transla- the starting point of s is P.Besides,we use the vector I to tion.Meanwhile,the tag movement can be regarded as the denote the polar axis AP,and use y to denote the angle particle movement only with the translation.Therefore,we between s and 1.Thus,if we use Ad to denote the projection investigate the relationship between the tag movement and of s on the polar axis 1,then Ad=lls]cosy. the rigid transformation of the tagged object,i.e.,translation, Note that,for any tag movement in the micro-movement, rotation and translation with rotation,respectively. its moving distance should be smaller than half-wavelength, i.e.,lsll≤≥≈l6.4cm.According to Eq.(1),by offsetting 4.4.1 Translation the constant diversity term,the phase variation A caused The translation means a linear movement that every point of the by s is as follows: device moves with the same displacement.Suppose a rigid body 402 ×2Ad=25x×2 silcos7 is attached with a tag array T,when the center of the rigid (2) body translates from position Ps to position Pe,each tag Ti Meanwhile,as 1.s sl cosy,according to Eq.(2), in the tag array has the same translation (ss) 1 1m·sA8. (3)Let and T be the coordinates of tagT Note that,,而 when the rigid body is at position P,and Pe,respectively, is a normalized vector of 1,it depends on the then: position of P relative to A.Assume s=(Az,Ay), 工i,e Ii.s +S. (7) (i,y),then,according to Eq.(3), yi,e yi.s x△x+△y=△， Fig.7(a)shows an example of the translation when the rigid (4) body is attached with a rectangle tag array. x+7=1. 4.4.2 Rotation Then,to compute the tag movement s=(△x,△y）accord- The rotation means a circular movement that the device rotates ing to the phase variations,we investigate their relation- around a rotation center.Suppose a rigid body is attached ships in the linear region and non-linear region,respectively. with a tag array T,when the rigid body rotates around a 4.3.1 Tag Movement in the Linear Region rotation center Pa by the angle of a,all the tags should In the linear region,the phase variations detected from have the same rotation angle.Specifically,letTand the two orthogonally deployed antennas are linear to the [i.e,yi.e]be the coordinates of tag Ti when the rigid body tag's moving distances along the two orthogonal axes,re- starts rotation and ends rotation,respectively,let(a,y)be spectively.E.g,as shown in Fig.4,antenna A detects the coordinates of rotation center Pa,then: the phase variation of the tag movement along the X-axis, Tie-Ta -R Ti,s-Ta (8) whereas antenna Ay detects the phase variations of the tag Yi,e Ya yi,s-Ya movement along the Y-axis.Let A0r and A0u be the phase cosa -sina variations from antenna Az and Ay,respectively,so the tag where R is a rotation matrix sina cosa representing movement s is computed as follows: the counter-clockwise rotation of angle a.Fig.7(b)shows - an example of the rotation when the rigid body is attached (5) with a rectangle tag array
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 5 ܺ ܻ Trajectory ܱ ܣ ܲ ܛ ܔ ߛ ݀߂ ܺ ܻ ܱ Fig. 5. Rigid transformation in continuous moving process ܺ ܻ ܣ ܲ ܛ ܔ ߛ ݀߂ ܱ ܣ ܲ ܛ ܔ ߛ ݀߂ ܺ ܻ ܱ Fig. 6. Relationship between tag movement and phase variation movement, it can be intuitively observed that, the rigid transformation of the tagged object, including the translation and rotation, can be derived from the movement of different tags. 4.3 Model of Tag Movement and Phase Variation According to Observation 1, the phase contours can be depicted as concentric circles with the antenna at the center. Thus, we can build a polar coordinate system by setting the center of the antenna as the origin. Then, given a tag movement s, we can further depict the relationship between the phase variation and the movement s in this polar coordinate system. As shown in Fig. 6, the antenna is deployed at position A, we use the vector s to denote the tag movement, the starting point of s is P. Besides, we use the vector l to denote the polar axis AP, and use γ to denote the angle between s and l. Thus, if we use ∆d to denote the projection of s on the polar axis l, then ∆d = ksk cos γ. Note that, for any tag movement in the micro-movement, its moving distance should be smaller than half-wavelength, i.e., ksk ≤ λ 2 ≈ 16.4cm. According to Eq. (1), by offsetting the constant diversity term, the phase variation ∆θ caused by s is as follows: ∆θ = 2π λ × 2∆d = 2π λ × 2 ksk cos γ. (2) Meanwhile, as l · s = klk · ksk cos γ, according to Eq. (2), l klk · s = λ 4π ∆θ. (3) Note that, l klk is a normalized vector of l, it depends on the position of P relative to A. Assume s = h∆x, ∆yi, l klk = hxl , yli, then, according to Eq. (3), xl∆x + yl∆y = λ 4π ∆θ, x 2 l + y 2 l = 1. (4) Then, to compute the tag movement s = h∆x, ∆yi according to the phase variations, we investigate their relationships in the linear region and non-linear region, respectively. 4.3.1 Tag Movement in the Linear Region In the linear region, the phase variations detected from the two orthogonally deployed antennas are linear to the tag’s moving distances along the two orthogonal axes, respectively. E.g., as shown in Fig. 4, antenna Ax detects the phase variation of the tag movement along the X-axis, whereas antenna Ay detects the phase variations of the tag movement along the Y -axis. Let ∆θx and ∆θy be the phase variations from antenna Ax and Ay, respectively, so the tag movement s is computed as follows: ∆x ∆y = λ 4π ∆θx λ 4π ∆θy . (5) 4.3.2 Tag Movement in the Non-linear Region In the non-linear region, since the corresponding phase variations are not linear to the tag movement, we need to figure out their relationship according to the geometric property. Given the phase variations ∆θx and ∆θy respectively collected from the two orthogonally deployed antennas Ax and Ay, according to Eq. (4), we have: xlx∆x + ylx∆y = λ 4π ∆θx, xly∆x + yly∆y = λ 4π ∆θy, (6) where hxlx , ylx i and xly , yly denote the normalized vector for the polar axis AP from the antenna Ax and Ay, respectively. Therefore, as long as the starting position of movement s , i.e, P, is known, the values of hxlx , ylx i and xly , yly can be figured out. Then, by solving the linear equations in Eq. (6), we can directly compute [∆x, ∆y] T . 4.4 Model of Rigid Motion Decomposition As aforementioned, during the continuous moving process of the rigid body, the micro-movement can be defined by the rigid transformation including the rotation and translation. Meanwhile, the tag movement can be regarded as the particle movement only with the translation. Therefore, we investigate the relationship between the tag movement and the rigid transformation of the tagged object, i.e., translation, rotation and translation with rotation, respectively. 4.4.1 Translation The translation means a linear movement that every point of the device moves with the same displacement. Suppose a rigid body is attached with a tag array T, when the center of the rigid body translates from position Ps to position Pe, each tag Ti in the tag array has the same translation S = sx, sy T . Let [xi,s, yi,s] T and [xi,e, yi,e] T be the coordinates of tag Ti when the rigid body is at position Ps and Pe, respectively, then: xi,e yi,e = xi,s yi,s + S. (7) Fig. 7(a) shows an example of the translation when the rigid body is attached with a rectangle tag array. 4.4.2 Rotation The rotation means a circular movement that the device rotates around a rotation center. Suppose a rigid body is attached with a tag array T, when the rigid body rotates around a rotation center Pa by the angle of α, all the tags should have the same rotation angle. Specifically, let [xi,s, yi,s] T and [xi,e, yi,e] T be the coordinates of tag Ti when the rigid body starts rotation and ends rotation, respectively, let (xa, ya) be the coordinates of rotation center Pa, then: xi,e − xa yi,e − ya = R xi,s − xa yi,s − ya , (8) where R is a rotation matrix cos α − sin α sin α cos α , representing the counter-clockwise rotation of angle α. Fig. 7(b) shows an example of the rotation when the rigid body is attached with a rectangle tag array.
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX.NO.XX,2020 oPa (a)Translation (b)Rotation (c)Translation with rotation Fig.7.Micro-movement decomposition 4.4.3 Translation with Rotation track the rigid motion of the tagged object,the design of the According to the definition of the rigid transformation,any arbi- rigid motion is shown in Fig.2.Detailed steps are as follows. trary rigid body motion can be decomposed into the combination of 1)Data Preprocessing:With RE-signals from the tag array, rotation and translation.Suppose a rigid body is attached with we extract the series of phase values for each tag from the a tag array T,when the center of the rigid body translates two antennas and segment them into the phase value for from the position P:to the position Pe,the rigid body also each snapshot.Then,we estimate the initial state of the rotates around a local rotation center Pa by the angle of a, tagged objects,including rotation state and rough position. the local rotation center has the same translation as the rigid 2)Movement Tracking:We derive the tag movement ac- body as well.Without loss of generality,we can model the cording to the phase variations,respectively,according to process of the rigid body motion into two successive steps the situations of the linear region and vast region(includ- one after the other,i.e.,performing the rotation first and ing both the linear region and non-linear region).Further, then the translation.Specifically,the rigid body first rotates we estimate the rigid transformation of the tagged object, around Pa by the angle of o,then it translates from position including the rotation and translation. P,to position Pe.According to Eq.(⑦and Eq.(⑧)，let 3)Movement Calibration:We detect the outliers of the and T be the coordinates of tag T when phase values from the tag array,by comparing the estimated the rigid body starts moving and ends moving,respectively, movement of each single tag with the estimated movement let (xa,ya)be the coordinates of local rotation center Pa of the tag array.Then,we eliminate the outlier(s)and re- when the rigid body starts moving,then: estimate the rigid transformation of the tagged object with the remaining tags for calibration. -Za=RTi-Ta +S. (9) Yi.e Ya Yi,8-ya According to Eq.(9),the movement of tag Ti,i.e. 5.1 Data Preprocessing for Rigid Motion Tracking [Ari,Ayi]T,can be decomposed into the following com- 5.1.1 Data Segmentation ponents: Due to the issues such as the multi-path effect and ambi- [△ Ti.e-Ti,s ent noises,the measured phase values may contain some △ ,e-,s =(R-IE,。xa] +S,(10) yi,s-Va fluctuations in the waveforms.Hence,after receiving the RF-signals from the tag array,we extract the series of phase where I is an identity matrix.Fig.7(c)shows an example values for each tag from the two antennas,and calibrate of the translation with rotation,when the rigid body is the phase values first.Specifically,due to the operation of attached with a rectangle tag array.Such rigid body motion mod in Eq.(1),the measured phase values are discontin- is equivalent to first rotating around Pa with angle o,i.e., uous.Thus,we stitch the phase values and remove the from the green array to the blue one,then translating from periodicity among the phase values.Besides,due to the P to Pe,i.e.,from the blue array to the yellow one. diversity term in Eq.(1),each tag has its own phase offset, so we measure the diversity term among tags in advance 5 DESIGN OF RIGID MOTION TRACKING and eliminate the tag diversity by offsetting the diversity We use a set of tags to track the rigid transformation of term.Further,we utilize the Kalman Filter to filter the cor- the tagged object,including the translation and the rotation. responding noises in the phase values.After that,suppose To support the tracking of small objects,we choose the there are n tags in the tag array,we segment the phases of small tag AZ9629 as the movement tag.Note that,during n tags from the two antennas into m snapshots,denoted the movement,the polarization angle of the tag relative to Θ1,1Θ1,2…Θ1，m the antenna changes as well,which can bring in additional as⊙= whereθi,=(日z,j,fu,i,》 phase offsets apart from the change of position.Whereas, Θn.1θn,2…θn,m for the movement tracking,we take each movement tag as means the phase values of tag Ti in the jth snapshot,0r.i.j a particle,and only focus on the phase offset caused by the and 0y.i.j represent the phase values from antenna A and position.Thus,we should reduce the phase offset due to the Ay,respectively.The time interval At for each snapshot change of the polarization angle during the movement of is usually set to a small value,e.g,At 200ms in our the tagged object.Empirically,we fix the tag orientation as implementation. shown in Fig.20,which is insensitive to the rotation due to the movement on the table. 5.1.2 Initial State Estimation RF-Dial extracts phase variations of the tag array re- According to Eq.(10),to compute the rigid transformation ceived by the orthogonally deployed RFID antenna pair to R,S,it is essential to determine the initial state of the
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 6 ௬ݏ ܲ௦ ܲ ௫ݏ ܺ ܻ ܱ (a) Translation ܺ ܻ ܲ ܲ௦ ߙ ܲ ܱ (b) Rotation ܲ ௬ݏ ௫ݏ ܲ௦(ܲ) ߙ ܺ ܻ ܱ (c) Translation with rotation Fig. 7. Micro-movement decomposition 4.4.3 Translation with Rotation According to the definition of the rigid transformation, any arbitrary rigid body motion can be decomposed into the combination of rotation and translation. Suppose a rigid body is attached with a tag array T, when the center of the rigid body translates from the position Ps to the position Pe, the rigid body also rotates around a local rotation center Pa by the angle of α, the local rotation center has the same translation as the rigid body as well. Without loss of generality, we can model the process of the rigid body motion into two successive steps one after the other, i.e., performing the rotation first and then the translation. Specifically, the rigid body first rotates around Pa by the angle of α, then it translates from position Ps to position Pe. According to Eq. (7) and Eq. (8), let [xi,s, yi,s] T and [xi,e, yi,e] T be the coordinates of tag Ti when the rigid body starts moving and ends moving, respectively, let (xa, ya) be the coordinates of local rotation center Pa when the rigid body starts moving, then: xi,e − xa yi,e − ya = R xi,s − xa yi,s − ya + S. (9) According to Eq. (9), the movement of tag Ti , i.e. [∆xi , ∆yi ] T , can be decomposed into the following components: ∆xi ∆yi = xi,e − xi,s yi,e − yi,s = (R − I) xi,s − xa yi,s − ya + S, (10) where I is an identity matrix. Fig. 7(c) shows an example of the translation with rotation, when the rigid body is attached with a rectangle tag array. Such rigid body motion is equivalent to first rotating around Pa with angle α, i.e., from the green array to the blue one, then translating from Ps to Pe, i.e., from the blue array to the yellow one. 5 DESIGN OF RIGID MOTION TRACKING We use a set of tags to track the rigid transformation of the tagged object, including the translation and the rotation. To support the tracking of small objects, we choose the small tag AZ9629 as the movement tag. Note that, during the movement, the polarization angle of the tag relative to the antenna changes as well, which can bring in additional phase offsets apart from the change of position. Whereas, for the movement tracking, we take each movement tag as a particle, and only focus on the phase offset caused by the position. Thus, we should reduce the phase offset due to the change of the polarization angle during the movement of the tagged object. Empirically, we fix the tag orientation as shown in Fig. 20, which is insensitive to the rotation due to the movement on the table. RF-Dial extracts phase variations of the tag array received by the orthogonally deployed RFID antenna pair to track the rigid motion of the tagged object, the design of the rigid motion is shown in Fig. 2. Detailed steps are as follows. 1) Data Preprocessing: With RF-signals from the tag array, we extract the series of phase values for each tag from the two antennas and segment them into the phase value for each snapshot. Then, we estimate the initial state of the tagged objects, including rotation state and rough position. 2) Movement Tracking: We derive the tag movement according to the phase variations, respectively, according to the situations of the linear region and vast region (including both the linear region and non-linear region). Further, we estimate the rigid transformation of the tagged object, including the rotation and translation. 3) Movement Calibration: We detect the outliers of the phase values from the tag array, by comparing the estimated movement of each single tag with the estimated movement of the tag array. Then, we eliminate the outlier(s) and reestimate the rigid transformation of the tagged object with the remaining tags for calibration. 5.1 Data Preprocessing for Rigid Motion Tracking 5.1.1 Data Segmentation Due to the issues such as the multi-path effect and ambient noises, the measured phase values may contain some fluctuations in the waveforms. Hence, after receiving the RF-signals from the tag array, we extract the series of phase values for each tag from the two antennas, and calibrate the phase values first. Specifically, due to the operation of mod in Eq. (1), the measured phase values are discontinuous. Thus, we stitch the phase values and remove the periodicity among the phase values. Besides, due to the diversity term in Eq. (1), each tag has its own phase offset, so we measure the diversity term among tags in advance and eliminate the tag diversity by offsetting the diversity term. Further, we utilize the Kalman Filter to filter the corresponding noises in the phase values. After that, suppose there are n tags in the tag array, we segment the phases of n tags from the two antennas into m snapshots, denoted as Θ = Θ1,1 Θ1,2 · · · Θ1,m · · · · · · · · · · · · Θn,1 Θn,2 · · · Θn,m , where Θi,j = hθx,i,j , θy,i,j i means the phase values of tag Ti in the jth snapshot, θx,i,j and θy,i,j represent the phase values from antenna Ax and Ay, respectively. The time interval ∆t for each snapshot is usually set to a small value, e.g, ∆t = 200ms in our implementation. 5.1.2 Initial State Estimation According to Eq. (10), to compute the rigid transformation R, S , it is essential to determine the initial state of the
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 and T;collected from antenna Ar,the estimation of Ari.j, i.e.,Aij can be computed as follows: (0z,i-0z,)×/(4r), l9z,i-6xπ (0x,i-9x,+2r)×/(4π).8x,i-0z,jT region (including both the linear region and non-linear (0z,i-0x,1+2x）×λ/(4r).0z,i-8x,<-元 region),respectively. Similarly,we can also compute the difference of the Eu- Linear Region:The linear region usually has an area clidean distance Ady.i.j for antenna Ay.Meanwhile,ac- of 0.6 x 0.6m2 in the intersection of the central beam cording to the position (x,y)and rotation angle of the tag regions for two orthogonally deployed antennas.If we set array B,we can also compute the theoretical value for the the operation area in this linear region,e.g.,a tabletop with difference of the Euclidean distance Adz.i.j and Ady.i.j. the size smaller than 0.6 x 0.6m2,then all the movement Hence,by leveraging the MMSE estimator,we can estimate can be controlled in this region.Suppose the tag array is the,,B}by finding the optimal set of parameters that located in the linear region,we first consider the relationship minimizes the squared errors for all pairs of tags. between the distance and the phase difference among tags for antenna Ar along the X-axis.For any two arbitrary tags (,y,B)"=arg min e'(,y,B), Ti and Tj in the tag array,if the distance between Ti and x,y,8 Tj along the X-axis,i.e.,Axi.j ri-xj,is smaller than half-wavelength A/2,according to the phase values of Ti e=∑∑（ad-△dP+(ad,-△d,P) 11
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 7 ܶଵ ȟݔଵଶ ܶଶ ܶଷ ܶସ ȟݕଵଶ ߚ ଷସݔȟ ߚ ȟݕଷସ ܶଵ ܶଶ ܶଷ ܶସ ߚ ݈ ݓ ݄ ߟ ܱ(ܲ) ܠܔ ܡܔ Fig. 8. Estimate the initial rotation angle of the object rigid body first. E.g., the matrix T = xi,s − xa yi,s − ya in Eq. (10) depends on the relative positions of tags in the tag array, i.e., the topology and rotation state of the tag array. Hence, since the rotation state of the tag array depends on both initial and subsequent rotation angles of the tag array, we need to estimate the initial rotation angle first. Without loss of generality, we use the rectangle tag array as an example to illustrate our method to compute the initial rotation angle for the tag array. As shown in Fig. 8, we set the rotation center Pa at the center of the rectangle, and set two orthogonal polar axes lx and ly according to the X and Y -axis in the global coordinate system. Each tag can be regarded as a particle point in the coordinate system, e.g., T1 ∼ T4. The initial rotation angle of the tag array, i.e., the angle between lx and PaT1, is β. Topology Matrix T. According to the rotation angle β, we can depict the matrix T = xi,s − xa yi,s − ya for each tag Ti . E.g., for the rectangle tag array in Fig.8, let kT1T4k = kT2T3k = h, kT1T2k = kT3T4k = w, kPaTik = l, 6 T1PaT2 = η, these parameters can be regarded as constants, as long as the topology of the tag array is fixed. Then, according to the relative positions of the tags in the rectangle tag array, [x1 − xa, y1 − ya] T = [l cos β, lsin β] T , [x2 − xa, y2 − ya] T = [l cos (β + η), lsin (β + η)]T , [x3 − xa, y3 − ya] T = [−l cos (β), −lsin (β)]T , [x4 − xa, y4 − ya] T = [−l cos (β + η), −lsin (β + η)]T . (11) As aforementioned in Section 4.2, the distances between tag pairs are linear/non-linear to the phase differences between tag pairs in the linear region/non-linear region, respectively. In the following, we provide solutions to figure out the rotation angle β for the linear region and the vast region (including both the linear region and non-linear region), respectively. Linear Region: The linear region usually has an area of 0.6 × 0.6m2 in the intersection of the central beam regions for two orthogonally deployed antennas. If we set the operation area in this linear region, e.g., a tabletop with the size smaller than 0.6 × 0.6m2 , then all the movement can be controlled in this region. Suppose the tag array is located in the linear region, we first consider the relationship between the distance and the phase difference among tags for antenna Ax along the X-axis. For any two arbitrary tags Ti and Tj in the tag array, if the distance between Ti and Tj along the X-axis, i.e., ∆xi,j = xi − xj , is smaller than half-wavelength λ/2, according to the phase values of Ti and Tj collected from antenna Ax, the estimation of ∆xi,j , i.e., ∆xbi,j can be computed as follows: ∆xbi,j = (θx,i − θx,j ) × λ/(4π), |θx,i − θx,j | π (θx,i − θx,j + 2π) × λ/(4π). θx,i − θx,j π (θx,i − θx,j + 2π) × λ/(4π). θx,i − θx,j < −π Similarly, we can also compute the difference of the Euclidean distance ∆dby,i,j for antenna Ay. Meanwhile, according to the position (x, y) and rotation angle of the tag array β, we can also compute the theoretical value for the difference of the Euclidean distance ∆dx,i,j and ∆dy,i,j . Hence, by leveraging the MMSE estimator, we can estimate the {x, y, β} by finding the optimal set of parameters that minimizes the squared errors for all pairs of tags. (x, y, β) ∗ = arg min x,y,β e 0 (x, y, β), e 0 = Xn i=1 Xn j=1 (∆dx,i,j − ∆dbx,i,j ) 2 + (∆dy,i,j − ∆dby,i,j ) 2
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 8 Specifically,to compute the parameters from the rotation Trajectory matrix Rt and translation matrix St,i.e.,ot,sr.t and sy.t we can expand Eq.(12)and obtain the following equations: (COS Ot-1)×6zt-sinat×6y,t+sz,t=△xi,t, (13) .sino×0z,t+(cos at-1)×dg,t+sy,t=△，t where 6r.t Ti.t-1 -Ta,t-1,6y,t Vi.t-1-ya,t-1.As we can obtain two such equations according to the phase values from a single tag received by two antennas,to solve the three Fig.9.Track continuous movement of rigid body via tag array unknown parameters at,s,t and sy.t from Eq.(13).At least two tags are essential to provide four equations to solve the 5.2 Movement Tracking unknown parameters.However,due to the issues such as 5.2.1 Derive the tag movement from the phase variation the multi-path effect and ambient noises,it is possible that To derive the tag movement from the phase variation col- the estimated tag movement might differ from the actual tag lected by the antenna pair,we also provide solutions for the movement in the rigid transformation Rt,St to a certain linear region and the vast region,respectively. extent.Hence,we need to deploy more tags in tag array to Linear Region:If we set the operation area in the linear track the rigid transformation in a more accurate manner. region,e.g.,a tabletop with the size smaller than 0.6x0.6m2 Suppose that we are able to collect the RF-signals from we can simply derive the tag movement [Azi,Ayi]T from n tags,given the parameters ot,s.t and sy.t,according the phase variation for each tag Ti according to Eq.(5). to Eq.(12),we can compute the theoretical tag movement Vast Region:As aforementioned in Section 5.1.2,the [△xi,t,△i,t小.Then,given the tag movement[△ti,t,△i,t] position of the tag array can be effectively estimated,then, derived from the phase variations,by leveraging the MMSE given the starting position of the tag movement,we can de- estimator,we aim to find the optimal solution af,s.t rive the tag movement [Azi,AyilT from the phase variation and s.t to minimize the difference between the theoret- for each tag Ti by solving Eq.(6). ical movement [Ari.t,Ayi.t]and the derived movement [△t.t,△，t 5.2.2 Track the continuous movement via the tag array The ultimate goal is to track the continuous movement of arg min (△x.t-△金.t)2+(△班.t-△.t)2). at:8s,t,8y,t i= the tagged object via the tag array.Hence,it is essential to track the rigid transformation R,S of the tagged objects for each micro-movement.As shown in Fig.9,we build a 5.3 Movement Calibration global 2D coordinate system according to the deployment Due to the issues such as the multi-path effect and mutual of the two mutually orthogonal antennas in Fig.4.Since interferences in the ambient environment,the phase values we only focus on the rigid transformation rather than the of some tags can be distorted to a certain extent,which absolute position of the tagged objects,the origin can be set might further impact the movement tracking of the rigid at any position.For simplicity,we can set the initial position body via the tag array.Specifically,the multi-path effect of the local rotation center of the tagged object as the origin. mainly comes from the signal reflections from the hand Hence,according to Eq.(10),during the moving pro- movement while using the tagged object as the HCI device; cess of the tagged object,the rigid transformation of the whereas the mutual interference mainly comes from the tagged object can be continuously estimated as follows:For interference of inductive coupling among adjacent tags. each snapshot t,suppose the rotation state in the previous Therefore,it is essential to detect the outliers of the phase snapshot t-1 is B-1,then we can estimate the matrix values,and further eliminate them to calibrate the estimated Ti,t-1 Ta,t-1 rigid transformation. Tt-1= from the previous snapshot t-1 yi.t-1-Ya.t-1 according to the tag array's topology and the rotation state. 5.3.1 Outlier Detection Then,according to the derived movement of tag Ti at Our solution is based on the observation that during the snapshot t,i.e.[Ari.t,Ayi.t]T,and the matrix Tt-1 from rigid transformation of the tagged object,if the phase values of the previous snapshot t-1,we can further figure out the one or more tags are severely distorted,then,for the tag with rotation matrix R and translation matrix St at snapshot t outlier phase values,the estimated displacement derived from from the following equation: the phase variations of this tag should be greatly different from the estimated displacement derived from the rigid transforma- A=Re-n)-1二at +St △，t (12) tion of the tag array.Specifically,for any specified tag Ti, yi,t-1 ya,t-1 assume that the estimated displacement is denoted with a After we figure out the rotation angle ot from the rotation vector Si =(Azi,Am),whereas the estimated displacement matrix Rt,we then update the rotation state in snapshot t derived from the rigid transformation is denoted with a as B:=Bt-1+at.This process iterates until the tagged vector si=(△x,△）.By comparing the cosine value object stops moving.For the initial rotation state of the of the vectorial angle c we can normalize the tag array,i.e.,the angle Bo,it can be figured out based on difference between the two vectors in the range [0,1].If ei the phase differences among the tags in the tag array,as is less than a threshold T,then we can identify tag Ti as a aforementioned in Section 5.1.2. candidate outlier
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 8 ܺ ܻ ߙ ଵߙ ܁ ଵ܁ Trajectory ܱ Fig. 9. Track continuous movement of rigid body via tag array 5.2 Movement Tracking 5.2.1 Derive the tag movement from the phase variation To derive the tag movement from the phase variation collected by the antenna pair, we also provide solutions for the linear region and the vast region, respectively. Linear Region: If we set the operation area in the linear region, e.g., a tabletop with the size smaller than 0.6×0.6m2 , we can simply derive the tag movement [∆xi , ∆yi ] T from the phase variation for each tag Ti according to Eq. (5). Vast Region: As aforementioned in Section 5.1.2, the position of the tag array can be effectively estimated, then, given the starting position of the tag movement, we can derive the tag movement [∆xi , ∆yi ] T from the phase variation for each tag Ti by solving Eq. (6). 5.2.2 Track the continuous movement via the tag array The ultimate goal is to track the continuous movement of the tagged object via the tag array. Hence, it is essential to track the rigid transformation R, S of the tagged objects for each micro-movement. As shown in Fig. 9, we build a global 2D coordinate system according to the deployment of the two mutually orthogonal antennas in Fig. 4. Since we only focus on the rigid transformation rather than the absolute position of the tagged objects, the origin can be set at any position. For simplicity, we can set the initial position of the local rotation center of the tagged object as the origin. Hence, according to Eq. (10), during the moving process of the tagged object, the rigid transformation of the tagged object can be continuously estimated as follows: For each snapshot t, suppose the rotation state in the previous snapshot t − 1 is βt−1, then we can estimate the matrix Tt−1 = xi,t−1 − xa,t−1 yi,t−1 − ya,t−1 from the previous snapshot t − 1 according to the tag array’s topology and the rotation state. Then, according to the derived movement of tag Ti at snapshot t, i.e. [∆xi,t, ∆yi,t] T , and the matrix Tt−1 from the previous snapshot t − 1, we can further figure out the rotation matrix Rt and translation matrix St at snapshot t from the following equation: ∆xi,t ∆yi,t = (Rt − I) xi,t−1 − xa,t−1 yi,t−1 − ya,t−1 + St. (12) After we figure out the rotation angle αt from the rotation matrix Rt, we then update the rotation state in snapshot t as βt = βt−1 + αt. This process iterates until the tagged object stops moving. For the initial rotation state of the tag array, i.e., the angle β0, it can be figured out based on the phase differences among the tags in the tag array, as aforementioned in Section 5.1.2. Specifically, to compute the parameters from the rotation matrix Rt and translation matrix St, i.e., αt, sx,t and sy,t, we can expand Eq. (12) and obtain the following equations: (cos αt − 1) × δx,t − sin αt × δy,t + sx,t = ∆xi,t, sin αt × δx,t + (cos αt − 1) × δy,t + sy,t = ∆yi,t, (13) where δx,t = xi,t−1 − xa,t−1, δy,t = yi,t−1 − ya,t−1. As we can obtain two such equations according to the phase values from a single tag received by two antennas, to solve the three unknown parameters αt, sx,t and sy,t from Eq. (13). At least two tags are essential to provide four equations to solve the unknown parameters. However, due to the issues such as the multi-path effect and ambient noises, it is possible that the estimated tag movement might differ from the actual tag movement in the rigid transformation Rt, St to a certain extent. Hence, we need to deploy more tags in tag array to track the rigid transformation in a more accurate manner. Suppose that we are able to collect the RF-signals from n tags, given the parameters αt, sx,t and sy,t, according to Eq. (12), we can compute the theoretical tag movement [∆xi,t, ∆yi,t]. Then, given the tag movement [∆xbi,t, ∆ybi,t] derived from the phase variations, by leveraging the MMSE estimator, we aim to find the optimal solution α ∗ t , s∗ x,t and s ∗ y,t to minimize the difference between the theoretical movement [∆xi,t, ∆yi,t] and the derived movement [∆xbi,t, ∆ybi,t]: arg min αt,sx,t,sy,t Xn i=1 (∆xi,t − ∆xbi,t) 2 + (∆yi,t − ∆ybi,t) 2 . 5.3 Movement Calibration Due to the issues such as the multi-path effect and mutual interferences in the ambient environment, the phase values of some tags can be distorted to a certain extent, which might further impact the movement tracking of the rigid body via the tag array. Specifically, the multi-path effect mainly comes from the signal reflections from the hand movement while using the tagged object as the HCI device; whereas the mutual interference mainly comes from the interference of inductive coupling among adjacent tags. Therefore, it is essential to detect the outliers of the phase values, and further eliminate them to calibrate the estimated rigid transformation. 5.3.1 Outlier Detection Our solution is based on the observation that during the rigid transformation of the tagged object, if the phase values of one or more tags are severely distorted, then, for the tag with outlier phase values, the estimated displacement derived from the phase variations of this tag should be greatly different from the estimated displacement derived from the rigid transformation of the tag array. Specifically, for any specified tag Ti , assume that the estimated displacement is denoted with a vector bsi = h∆xbi , ∆ybii, whereas the estimated displacement derived from the rigid transformation is denoted with a vector si = h∆xi , ∆yii. By comparing the cosine value of the vectorial angle i = bsi·si kbsikksik , we can normalize the difference between the two vectors in the range [0, 1]. If i is less than a threshold τ , then we can identify tag Ti as a candidate outlier.
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 5.3.2 Outlier Elimination Antenna As there might exist one or more outliers for the phases of tags,one or more outliers can distort the overall rigid transformation to a certain extent,to effectively eliminate 120cm nple index 300 the outliers,our solution is designed as follows:We first compute ei for each tag Ti in the tag array.Among the A RSSI candidate outliers with eir or half of Fig.10.RSSI deviation during swipe from left to right the total tags are eliminated.At last,we use the value of R,S after the outlier elimination as the final calibrated swipe from left to right on the tag,the RSSI increases from left to result for the rigid transformation. center and then decreases from center to right,the RSSI variation The MMSE function can be solved quickly with ad- of the swipe process is relatively symmetrical around the tag's vanced algorithms,i.e.,Gauss-Newton iteration method. center and forms an -wave pattern.As shown in Fig.10(a) When the multi-path is very serious,it is possible that the we perform the swipe with two settings,the RSSI is plotted residual of the MMSE function is still very large,thereby, in Fig.10(b),where the top figure is the raw RSSI from RFID, the motion tracking fails or has the poor accuracy.Here, the bottom figure is the RSSI values with different touch po- the serious multi-path effect often happens when there are sitions by combining OptiTrack data and RFID data.When moving people around the monitoring area within 60cm the finger presses down on the left of the tag,RSSI decreases The user's operating hand is usually considered to have about 20dB.RSSI has larger values when the touch position much influence on the RF-signals,but during the operation, approaches the tag center,the maximum RSSI value is just its influence on different tags is similar.As we leverage the 1dB smaller than the untouched situation.The whole RSSI phase variation of a tag or phase difference between tags,as variation forms an !-wave.This is because that the human long as the signal distortion is steady,we can still achieve skin can be modeled as an equivalent impedance,when the the accurate motion tracking.While for the serious multi- user touches the tag,the capacitive coupling happens that path effect,it is the main drawback for almost all wireless influences the backscattered signals from the tag.As our tag sensing solutions,including our work.It seems a pity that has the dipole antenna,theoretically there exists a mirror the wireless sensing is sensitive to the environment,but position on the other antenna of the tag that has the same many researchers have put much effort into extracting the coupling effect for any touch position.Note that,the multi- clean signal combating the environment interference [23], path effect is different at different positions,so the signals which is beyond the scope of our work.In this paper,we aim of two mirror positions will not be exactly the same,but the to provide a light-weight and functional human-computer RSSI variation pattern seems symmetrical on the whole. interaction solution,so we simply have the assumption Observation 3:The RSSI deviation keeps steady regardless that there is no moving object within the monitoring area of the tag's position and orientation or even the user.Here, except the tagged object.In this manner,the received signals RSSI deviation refers to the RSSI difference between the will not too bad and can be used for the motion tracking. RSSI value when a tag is touched and the RSSI value when However,it does not mean that our solution cannot tolerant the tag is untouched,denoted as ARSSI.The absolute RSSI the environmental factors at all.We tested the heavy multi- value is related to the position or orientation of the tag path situation when there are many reflection things within relative to the antenna as shown in Fig.10,but the RSSI the monitoring area,and results show that our solution can variation patterns are similar.Let the stable RSSI value work well in this situation,details are shown in Section 9.2. just before touch be the reference value (denoted as C), so the RSSI deviation is calculated as:ARSSI RSSI-C, 6 TOUCH GESTURE DETECTION the ARSSI pattern during the swipe is relatively steady Different from the rigid motion detection where we view a despite the settings.In particular,we explore more about tag as particle and small tags are preferred,in this section, the ARSSI pattern.As illustrated in Fig.11(a),a tag is in we use a linear dipole tag as the fine-grained touch interface. front of the antenna's center about 80cm with its orientation parallel to the antenna plane(=0),a user faces the antenna 6.1 RSSI Deviation during Touch Gesture and stays behind the tag of 40cm (d=40cm)by default. We use a linear dipole tag E51 as the touch interface to Through adjusting different positions,orientations or users, explore the signal characteristics due to the touch gesture. we collect tag signals during the swipe,results are plotted in We put a tag on a table in front of an antenna.The antenna Fig.11(b)-11(d).It is found that the RSSI deviation patterns has the same height as the table.We perform the swipe on at different settings have the high similarity.The n-waves the tag from left to right with different settings.We leverage almost coincide with each other,especially in terms of the OptiTrack to track the finger position.By synchronizing distance and users.For different angles,the similarity of the finger trace from OptiTrack and the received RF-signals wave patterns is a bit smaller than that of distance or user, based on timestamps,we can derive the RSSI variation of but it is still very high.It is reasonable that the capacitive the tag with different touch positions. coupling effect accounts for the major signal variation when Observation 2:The received signal strength(RSSI)of a tag a tag is touched as the impedance of different users is similar changes significantly when a user touches the tag.During the for the tag,so the RSSI deviation keeps steady for different
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 9 5.3.2 Outlier Elimination As there might exist one or more outliers for the phases of tags, one or more outliers can distort the overall rigid transformation to a certain extent, to effectively eliminate the outliers, our solution is designed as follows: We first compute i for each tag Ti in the tag array. Among the candidate outliers with i < τ , we select the tag with the minimum value of i . We then eliminate this tag and recalculate the rigid transformation R, S according to the phase values of the remaining tags. We repeat the above process until all the remaining tags have i ≥ τ or half of the total tags are eliminated. At last, we use the value of R, S after the outlier elimination as the final calibrated result for the rigid transformation. The MMSE function can be solved quickly with advanced algorithms, i.e., Gauss-Newton iteration method. When the multi-path is very serious, it is possible that the residual of the MMSE function is still very large, thereby, the motion tracking fails or has the poor accuracy. Here, the serious multi-path effect often happens when there are moving people around the monitoring area within 60cm. The user’s operating hand is usually considered to have much influence on the RF-signals, but during the operation, its influence on different tags is similar. As we leverage the phase variation of a tag or phase difference between tags, as long as the signal distortion is steady, we can still achieve the accurate motion tracking. While for the serious multipath effect, it is the main drawback for almost all wireless sensing solutions, including our work. It seems a pity that the wireless sensing is sensitive to the environment, but many researchers have put much effort into extracting the clean signal combating the environment interference [23], which is beyond the scope of our work. In this paper, we aim to provide a light-weight and functional human-computer interaction solution, so we simply have the assumption that there is no moving object within the monitoring area except the tagged object. In this manner, the received signals will not too bad and can be used for the motion tracking. However, it does not mean that our solution cannot tolerant the environmental factors at all. We tested the heavy multipath situation when there are many reflection things within the monitoring area, and results show that our solution can work well in this situation, details are shown in Section 9.2. 6 TOUCH GESTURE DETECTION Different from the rigid motion detection where we view a tag as particle and small tags are preferred, in this section, we use a linear dipole tag as the fine-grained touch interface. 6.1 RSSI Deviation during Touch Gesture We use a linear dipole tag E51 as the touch interface to explore the signal characteristics due to the touch gesture. We put a tag on a table in front of an antenna. The antenna has the same height as the table. We perform the swipe on the tag from left to right with different settings. We leverage OptiTrack to track the finger position. By synchronizing the finger trace from OptiTrack and the received RF-signals based on timestamps, we can derive the RSSI variation of the tag with different touch positions. Observation 2: The received signal strength (RSSI) of a tag changes significantly when a user touches the tag. During the Left (0cm) Right (9.5cm) Center Antenna Setting1 Setting2 120cm 60cm 20cm Left (0cm) Right (9.5cm) (a) Swipe (top view) 0 2 4 6 8 10 Touch position (cm) -60 -40 RSSI (dBm) Setting1 Setting2 0 150 300 450 Sample index -60 -40 RSSI (dBm) Swipe process Reference RSSI RSSI (b) RSSI deviation Fig. 10. RSSI deviation during swipe from left to right swipe from left to right on the tag, the RSSI increases from left to center and then decreases from center to right, the RSSI variation of the swipe process is relatively symmetrical around the tag’s center and forms an Ω-wave pattern. As shown in Fig. 10(a), we perform the swipe with two settings, the RSSI is plotted in Fig. 10(b), where the top figure is the raw RSSI from RFID, the bottom figure is the RSSI values with different touch positions by combining OptiTrack data and RFID data. When the finger presses down on the left of the tag, RSSI decreases about 20dB. RSSI has larger values when the touch position approaches the tag center, the maximum RSSI value is just 1dB smaller than the untouched situation. The whole RSSI variation forms an Ω-wave. This is because that the human skin can be modeled as an equivalent impedance, when the user touches the tag, the capacitive coupling happens that influences the backscattered signals from the tag. As our tag has the dipole antenna, theoretically there exists a mirror position on the other antenna of the tag that has the same coupling effect for any touch position. Note that, the multipath effect is different at different positions, so the signals of two mirror positions will not be exactly the same, but the RSSI variation pattern seems symmetrical on the whole. Observation 3: The RSSI deviation keeps steady regardless of the tag’s position and orientation or even the user. Here, RSSI deviation refers to the RSSI difference between the RSSI value when a tag is touched and the RSSI value when the tag is untouched, denoted as ∆RSSI. The absolute RSSI value is related to the position or orientation of the tag relative to the antenna as shown in Fig. 10, but the RSSI variation patterns are similar. Let the stable RSSI value just before touch be the reference value (denoted as C), so the RSSI deviation is calculated as: ∆RSSI = RSSI−C, the ∆RSSI pattern during the swipe is relatively steady despite the settings. In particular, we explore more about the ∆RSSI pattern. As illustrated in Fig. 11(a), a tag is in front of the antenna’s center about 80cm with its orientation parallel to the antenna plane (γ=0◦ ), a user faces the antenna and stays behind the tag of 40cm (d=40cm) by default. Through adjusting different positions, orientations or users, we collect tag signals during the swipe, results are plotted in Fig. 11(b)-11(d). It is found that the RSSI deviation patterns at different settings have the high similarity. The Ω-waves almost coincide with each other, especially in terms of the distance and users. For different angles, the similarity of Ω- wave patterns is a bit smaller than that of distance or user, but it is still very high. It is reasonable that the capacitive coupling effect accounts for the major signal variation when a tag is touched as the impedance of different users is similar for the tag, so the RSSI deviation keeps steady for different
IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 10 60 cm 0 Touch position(cm) Touch position(cm) Touch position(cm) (a)Swipe with different settings (b)Swipe with different distances (c)Swipe with different angles (d)Swipe with different users Fig.11.RSSI deviation during swipe Antenna Transmitted Error of toucl signal posifon Is de 30 de -45de Distance difference 2 6 ，Reflection Touch position(cm)】 Touch position(cm) Touch position(cm) signal Fig.12.RSSI deviation during Fig.13.Error of touch position Fig.14.Phase deviation during Fig.15.Distance difference during swipe with different scenarios swipe with different angles swipe distances and users.While for different angles,the tag is not deviation as the phase difference between the phase values parallel to the antenna plane,the hand would move away when a tag is or not touched,denoted as Aphase.When from or towards the antenna,thus the power of reflection the tag orientation gets changed,the phase deviation does signals from the hand decreases with the larger distance or not keep steady.Fig.14 plots the phase variation patterns increases with the smaller distance.But thankfully,as shown during the swipe when putting the tag with different angles, in Fig.11(c),such interference is acceptable as long as a tag and these patterns are different from each other.It means can be always activated during the whole swipe.When the that if using the pattern of 0 as the template,we cannot angle gets larger,the tag receives less power from signals, determine the accurate touch position when the angle is 45. it is easy for the tag not to be activated.In our settings,if This phenomenon is likely to be caused by the difference of increasing the angle after 60,the tag cannot continuously the reflection signals from the moving hand.As shown in response to the reader during the swipe.As we deploy one Fig.15,the signal received by the tag contains not only the antenna pair orthogonally to track the rigid motion in Fig.1, transmitted signals from the antenna,but also the reflection the polarization angle between the tag to one of the antenna signals from the surrounding objects,among which the pair is within 45 in the central scanning area,the tag can be hand touching the tag is the main reflector.When the tag is interrogated by at least one antenna,thus we think the RSSI not parallel to the antenna plane,the distance between the deviation is insensitive to the angle. hand and the antenna changes during the swipe process, Although in one scenario,i.e.,a certain tagged object in so the reflection signal from the hand to the tag changes a certain environment,the RSSI deviation is irrelevant to the a bit as well.The larger angle between the tag orientation distance or the user and insensitive to the angle,the static and the antenna plane,the larger distance difference of the multi-path effect in different scenarios differs and influences reflection signal from the moving hand.As the range of the the RSSI variation pattern,as shown in Fig.12.In Scenario 1, distance variation is just several centimeters,the strength of a tag is attached to a paper box in an office room by default, the reflection signal can be viewed unchanged.However,as while we attach the tag to a foam box in Scenario 2 and do mentioned above,the phase is very sensitive to the distance, the experiment in a hall in Scenario 3.It is observed that the so the slight distance difference of the moving hand can ranges of RSSI patterns are different.If using the smoothed lead to the significant phase difference of the reflection pattern of Scenario 1 as template,we cannot always locate signal,which further affects the tag signals collected by the the touch position accurately in other scenarios,as shown antenna.If using the phase,we need to determine the tag in Fig.13.Also,as the RSSI deviation pattern is symmetric, orientation first,then generate the pattern templates for each we would get two candidate touch positions centered of the tag orientation,adding much burden for the training and tag center based on one RSSI deviation value.To tackle this detection.Meanwhile,the amplitude of the phase variance ambiguity,we propose to use two tag antennas for different due to the touch is much smaller than the RSSI variance,so functions,so we can get the unique touch position,details the RSSI deviation is more significant and more robust to are shown in Section 7.Therefore,to ensure the accuracy of the environment interference.Therefore,we select the RSSI the touch position,we need to generate templates for each deviation to detect the touch gesture. scenario,but for the same scenario,one general template is satisfactory for different distances,angles or users. 6.3 Model of RSSI Deviation due to Touch Gesture The power received by the receiver from the transmitter 6.2 Phase Deviation during Touch Gesture can be described by the Friis equation [32].Denote the Compared with the RSSI,the phase is a more sensitive received and transmit power as PR and Pr,the receiver gain parameter of backscattered signals.Similarly,define phase and transmitter gain as GR and Gr,the polarization angle
IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 10 80cm � � � 100 200 � � 60 � � Antenna � (a) Swipe with different settings 0 2 4 6 8 10 Touch position (cm) -30 -20 -10 0 RSSI (dBm) 60 cm 80 cm 100 cm 120 cm (b) Swipe with different distances 0 2 4 6 8 10 Touch position (cm) -30 -20 -10 0 RSSI (dBm) 0 deg 15 deg 30 deg 45 deg 60 deg (c) Swipe with different angles 0 2 4 6 8 10 Touch position (cm) -30 -20 -10 0 RSSI (dBm) User1 User2 User3 (d) Swipe with different users Fig. 11. RSSI deviation during swipe 0 2 4 6 8 10 Touch position (cm) -30 -20 -10 0 RSSI (dBm) Scenario 1 Scenario 2 Scenario 3 Fig. 12. RSSI deviation during swipe with different scenarios 0 2 4 6 8 10 Touch position (cm) -30 -20 -10 0 RSSI (dBm) Template Test Error of touch position True position Candidate position 1 Candidate position 2 Fig. 13. Error of touch position distances and users. While for different angles, the tag is not parallel to the antenna plane, the hand would move away from or towards the antenna, thus the power of reflection signals from the hand decreases with the larger distance or increases with the smaller distance. But thankfully, as shown in Fig. 11(c), such interference is acceptable as long as a tag can be always activated during the whole swipe. When the angle gets larger, the tag receives less power from signals, it is easy for the tag not to be activated. In our settings, if increasing the angle after 60◦ , the tag cannot continuously response to the reader during the swipe. As we deploy one antenna pair orthogonally to track the rigid motion in Fig. 1, the polarization angle between the tag to one of the antenna pair is within 45◦ in the central scanning area, the tag can be interrogated by at least one antenna, thus we think the RSSI deviation is insensitive to the angle. Although in one scenario, i.e., a certain tagged object in a certain environment, the RSSI deviation is irrelevant to the distance or the user and insensitive to the angle, the static multi-path effect in different scenarios differs and influences the RSSI variation pattern, as shown in Fig. 12. In Scenario 1, a tag is attached to a paper box in an office room by default, while we attach the tag to a foam box in Scenario 2 and do the experiment in a hall in Scenario 3. It is observed that the ranges of RSSI patterns are different. If using the smoothed pattern of Scenario 1 as template, we cannot always locate the touch position accurately in other scenarios, as shown in Fig. 13. Also, as the RSSI deviation pattern is symmetric, we would get two candidate touch positions centered of the tag center based on one RSSI deviation value. To tackle this ambiguity, we propose to use two tag antennas for different functions, so we can get the unique touch position, details are shown in Section 7. Therefore, to ensure the accuracy of the touch position, we need to generate templates for each scenario, but for the same scenario, one general template is satisfactory for different distances, angles or users. 6.2 Phase Deviation during Touch Gesture Compared with the RSSI, the phase is a more sensitive parameter of backscattered signals. Similarly, define phase 0 2 4 6 8 10 Touch position (cm) -5 -4 -3 -2 -1 0 Phase (rad.) 0 deg 15 deg 30 deg 45 deg Fig. 14. Phase deviation during swipe with different angles Antenna Setting1 Left (0cm) Right (9.5cm) Distance difference Transmitted signal Reflection signal Fig. 15. Distance difference during swipe deviation as the phase difference between the phase values when a tag is or not touched, denoted as ∆phase. When the tag orientation gets changed, the phase deviation does not keep steady. Fig. 14 plots the phase variation patterns during the swipe when putting the tag with different angles, and these patterns are different from each other. It means that if using the pattern of 0 ◦ as the template, we cannot determine the accurate touch position when the angle is 45◦ . This phenomenon is likely to be caused by the difference of the reflection signals from the moving hand. As shown in Fig. 15, the signal received by the tag contains not only the transmitted signals from the antenna, but also the reflection signals from the surrounding objects, among which the hand touching the tag is the main reflector. When the tag is not parallel to the antenna plane, the distance between the hand and the antenna changes during the swipe process, so the reflection signal from the hand to the tag changes a bit as well. The larger angle between the tag orientation and the antenna plane, the larger distance difference of the reflection signal from the moving hand. As the range of the distance variation is just several centimeters, the strength of the reflection signal can be viewed unchanged. However, as mentioned above, the phase is very sensitive to the distance, so the slight distance difference of the moving hand can lead to the significant phase difference of the reflection signal, which further affects the tag signals collected by the antenna. If using the phase, we need to determine the tag orientation first, then generate the pattern templates for each tag orientation, adding much burden for the training and detection. Meanwhile, the amplitude of the phase variance due to the touch is much smaller than the RSSI variance, so the RSSI deviation is more significant and more robust to the environment interference. Therefore, we select the RSSI deviation to detect the touch gesture. 6.3 Model of RSSI Deviation due to Touch Gesture The power received by the receiver from the transmitter can be described by the Friis equation [32]. Denote the received and transmit power as PR and PT , the receiver gain and transmitter gain as GR and GT , the polarization angle
