IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 RF-Dial:Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFId Tags Yanling Bu,Student Member,IEEE,Lei Xie,Member,IEEE,Yinyin Gong,Chuyu Wang,Member,IEEE, Lei Yang,Member,IEEE,Jia Liu,Member,IEEE,and Sanglu Lu,Member,IEEE Abstract-With the rising of demands for novel human-computer interaction approaches in the 2D plane,a number of intelligent devices come into being.For example,Microsoft Surface Dial supports simple clicks and rotations for the interaction with computer. However,these approaches are dedicated devices,and they might require batteries or have limited functions.In this paper,we propose RF-Dial to realize a light-weight,battery-free and functional 2D human-computer interaction solution via commercial off-the-shelf (COTS)passive RFID tags.What RF-Dial shines is that it can easily turn an ordinary object,e.g.,a board eraser,into an intelligent interaction device.By deploying a tag array on the side face of the object together with a dipole tag on the top face,RF-Dial cannot only track the rigid motion of the object but also detect the touch gesture of a user on the surface of the object,including translation, rotation,click,press and hold,and swipe.To do the motion tracking,RF-Dial builds a phase-based model that captures the translation and the rotation of the tagged object simultaneously,by jointly exploiting the information of phase variations and the topology of the tag array.To detect the touch gesture,RF-Dial builds an RSSI-based model that uses the impact of the touching finger on the tag antenna's impedance to estimate the touch position in real time,which is robust to environmental factors like position or orientation.We implemented a prototype of RF-Dial with commodity RFID devices.Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking,with a small error of 0.6cm for the translation tracking,and a small error of 1.9 degrees for the rotation estimation. Besides,RF-Dial can also detect the touch gesture accurately,as the 90 percent of touch position errors are less than 2.09mm. Index Terms-RFID,human-computer interaction,tag array,translation,rotation,coupling effect,touch gesture. ◆ INTRODUCTION TN modern times,the widely used approaches for the novel HCI designs [2-18].RFID can even work in the non- human-computer interaction (HCI)are operated in the line-of-sight situation due to its backscatter communication. 2D plane,like the touch screen and the mouse.By moving Therefore,we hope to use RFID to answer such a question: or stroking these interaction devices,users can access the "Is it possible to design a battery-free and light-weight solution objects in the computer and manipulate them conveniently.to the 2D human-computer interaction,thereby even an ordinary With the rise of the computer aided art design and other object can be easily turned into an intelligent interaction device?" novel applications,a number of intelligent devices have In this paper,we propose RF-Dial to realize a novel come into being as the response to the demand for brand-2D human-computer interaction solution via COTS passive new 2D interaction solutions.For example,Microsoft Sur- RFID tags.We attach a tag array to the side face together face Dial [1]emerged in 2016,supporting simple clicks and with one tag on the top face of an object,denote them as rotations for the natural and friendly interaction. movement tags and the touch tag,respectively.As shown The latest HCI approaches are mainly based on the in Fig.1,we deploy two RFID antennas orthogonally to computer vision or sensors.For computer vision-based realize our vision.Specifically,we continuously track the approaches,they use cameras to monitor the movement rigid motion of the tagged object with movement tags,in- of limbs or fingers.However,they are mainly limited by cluding the translation and the rotation simultaneously,and privacy concerns,the light condition and the viewing an- detect the touch gesture with the touch tag,including the gle.For sensor-based approaches,they use commercial- click,the press and hold,and the swipe.In this way,an off-the-shelf (COTS)sensors like inertial sensors to track ordinary object such as a candy box can be turned into an the movement of devices.Their main constraints are the intelligent interaction device.For example,we can realize limited battery life and the high hardware cost.Thankfully, an functional drawing application with RF-Dial.It tracks RFID provides the battery-free sensing technology to enable the translation of movement tags to draw lines,and adjusts the line color automatically by the rotation just during the drawing process;The touch tag functions as buttons and Yanling Bu,Lei Xie,Yinyin Gong,Chuyu Wang,Jia Liu,and Sanglu Lu are with the State Key Laboratory for Novel Softiare Technology,Nanjing sliders,receiving the user's commands,i.e.,adjusting the University,China. line color and width,as shown in the case study (Sec- E-mail: yanling@smail.nju.edu.cn, Ixie@nju.edu.cn, yy-tion 9.4).Technically,based on RF-signals from movement gong@dislab.nju.edu.cn, chuyu@nju.edu.cn, jialiu@nju.edu.cn, tags,we build a rigid transformation model to reflect the sanglu@nju.edu.cn. Lei Yang is with the Department of Computing,The Hong Kong Poly- relationship between the motion of the tagged object and technic University,Hong Kong,China. the corresponding phase variations of each movement tag in E-mail:young @tagsys.org. the tag array.As the movement tags form a tag array with .Lei Xie is the corresponding author. the fixed topology,we can derive the translation and theIEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 1 RF-Dial: Rigid Motion Tracking and Touch Gesture Detection for Interaction via RFID Tags Yanling Bu, Student Member, IEEE, Lei Xie, Member, IEEE, Yinyin Gong, Chuyu Wang, Member, IEEE, Lei Yang, Member, IEEE, Jia Liu, Member, IEEE, and Sanglu Lu, Member, IEEE Abstract—With the rising of demands for novel human-computer interaction approaches in the 2D plane, a number of intelligent devices come into being. For example, Microsoft Surface Dial supports simple clicks and rotations for the interaction with computer. However, these approaches are dedicated devices, and they might require batteries or have limited functions. In this paper, we propose RF-Dial to realize a light-weight, battery-free and functional 2D human-computer interaction solution via commercial off-the-shelf (COTS) passive RFID tags. What RF-Dial shines is that it can easily turn an ordinary object, e.g., a board eraser, into an intelligent interaction device. By deploying a tag array on the side face of the object together with a dipole tag on the top face, RF-Dial cannot only track the rigid motion of the object but also detect the touch gesture of a user on the surface of the object, including translation, rotation, click, press and hold, and swipe. To do the motion tracking, RF-Dial builds a phase-based model that captures the translation and the rotation of the tagged object simultaneously, by jointly exploiting the information of phase variations and the topology of the tag array. To detect the touch gesture, RF-Dial builds an RSSI-based model that uses the impact of the touching finger on the tag antenna’s impedance to estimate the touch position in real time, which is robust to environmental factors like position or orientation. We implemented a prototype of RF-Dial with commodity RFID devices. Extensive experiments show that RF-Dial achieves an accurate rigid motion tracking, with a small error of 0.6cm for the translation tracking, and a small error of 1.9 degrees for the rotation estimation. Besides, RF-Dial can also detect the touch gesture accurately, as the 90 percent of touch position errors are less than 2.09mm. Index Terms—RFID, human-computer interaction, tag array, translation, rotation, coupling effect, touch gesture. ✦ 1 INTRODUCTION I N modern times, the widely used approaches for the human-computer interaction (HCI) are operated in the 2D plane, like the touch screen and the mouse. By moving or stroking these interaction devices, users can access the objects in the computer and manipulate them conveniently. With the rise of the computer aided art design and other novel applications, a number of intelligent devices have come into being as the response to the demand for brand￾new 2D interaction solutions. For example, Microsoft Sur￾face Dial [1] emerged in 2016, supporting simple clicks and rotations for the natural and friendly interaction. The latest HCI approaches are mainly based on the computer vision or sensors. For computer vision-based approaches, they use cameras to monitor the movement of limbs or fingers. However, they are mainly limited by privacy concerns, the light condition and the viewing an￾gle. For sensor-based approaches, they use commercial￾off-the-shelf (COTS) sensors like inertial sensors to track the movement of devices. Their main constraints are the limited battery life and the high hardware cost. Thankfully, RFID provides the battery-free sensing technology to enable • Yanling Bu, Lei Xie, Yinyin Gong, Chuyu Wang, Jia Liu, and Sanglu Lu are with the State Key Laboratory for Novel Software Technology, Nanjing University, China. E-mail: yanling@smail.nju.edu.cn, lxie@nju.edu.cn, yy￾gong@dislab.nju.edu.cn, chuyu@nju.edu.cn, jialiu@nju.edu.cn, sanglu@nju.edu.cn. • Lei Yang is with the Department of Computing, The Hong Kong Poly￾technic University, Hong Kong, China. E-mail: young@tagsys.org. • Lei Xie is the corresponding author. novel HCI designs [2–18]. RFID can even work in the non￾line-of-sight situation due to its backscatter communication. Therefore, we hope to use RFID to answer such a question: “Is it possible to design a battery-free and light-weight solution to the 2D human-computer interaction, thereby even an ordinary object can be easily turned into an intelligent interaction device?”. In this paper, we propose RF-Dial to realize a novel 2D human-computer interaction solution via COTS passive RFID tags. We attach a tag array to the side face together with one tag on the top face of an object, denote them as movement tags and the touch tag, respectively. As shown in Fig. 1, we deploy two RFID antennas orthogonally to realize our vision. Specifically, we continuously track the rigid motion of the tagged object with movement tags, in￾cluding the translation and the rotation simultaneously, and detect the touch gesture with the touch tag, including the click, the press and hold, and the swipe. In this way, an ordinary object such as a candy box can be turned into an intelligent interaction device. For example, we can realize an functional drawing application with RF-Dial. It tracks the translation of movement tags to draw lines, and adjusts the line color automatically by the rotation just during the drawing process; The touch tag functions as buttons and sliders, receiving the user’s commands, i.e., adjusting the line color and width, as shown in the case study (Sec￾tion 9.4). Technically, based on RF-signals from movement tags, we build a rigid transformation model to reflect the relationship between the motion of the tagged object and the corresponding phase variations of each movement tag in the tag array. As the movement tags form a tag array with the fixed topology, we can derive the translation and the