RF-Brush:3D Human-Computer Interaction via Linear Tag Array Yinyin Gong,Lei Xie,Chuyu Wang,Yanling Bu,Sanglu Lu State Key Laboratory for Novel Software Technology,Nanjing University Email:[yygong,wangcyu217)@dislab.nju.edu.cn,[Ixie,sanglu)@nju.edu.cn,yanling@smail.nju.edu.cn Abstract-Nowadays,novel approaches of 3D human-computer interaction have enabled the capability of manipulating in the 3D Tag 3D Orientation Array space rather than 2D space.For example,Microsoft Surface Pen leverages the embedded sensors to sense the 3D manipulations such as inclining the pen to get bolder handwriting.In this paper,we propose RF-Brush,a battery-free and light-weight solution for 3D human-computer interaction based on RFID,by simply attaching a linear RFID tag array onto the linear shaped object like a brush.RF-Brush senses the 3D orientation and 2D movement of the linear shaped object,when the human subject is drawing with this object in the 3D space.Here,the 3D orientation 21 Operating Plane refers to the relative orientation of the linear shaped object to the operating plane,whereas the 2D movement refers to the moving trace in the 2D operating plane.In this way,we are able to Fig.1:Illustration of applications scenario of RF-Brush. transform an ordinary linear shaped object like a brush or pen need extensive training process.They either are sensitive to to an intelligent HCI device.Particularly,we build two geometric the change of background and light condition,or only pro- models to depict the relationship between the RF-signal and the 3D orientation as well as 2D movement,respectively.Based vide coarse-grained classification manipulation.Sensor-based on the geometric model,we propose the linear tag array-based approaches [6]usually can accurately track the motion of the HCI solution,implemented a prototype system,and evaluated device in the 3D space based on the embedded IMU.But they the performance in real environment.The experiments show usually suffer from the limited battery life.As for RFID-based that RF-Brush achieves an average error of 5.7 and 8.6 of elevation and azimuthal angle,respectively,and an average error approaches [7].[8].most of them are focusing on interacting of 3.8cm and 4.2cm in movement tracking along X-axis and Y. with computer in 2D space.Therefore,according to the above axis,respectively.Moreover,RF-Brush achieves 89%in letter considerations,in this paper,we aim to provide a battery-free recognition accuracy. and more natural 3D human-computer interaction by tracking I.INTRODUCTION the fine-grained 3D motion based on cheap ubiquitous objects. With the ubiquity of sensing techniques,human-computer As Fig.I shows,we interact with computer not only by the interaction (HCl)has drawn more and more attention recently, moving trace in 2D space,but also the orientation.Here,the and novel approaches are required to provide a more natural orientation is able to reflect the thickness of lines,and the 2D interaction.Currently,due to the wide utilization of the smart moving trace reflects what we write finally devices (e.g.,smart phone),the touch screen has become the With the development of RFID technique,RFID tag has most popular HCI interaction approach.However,by using the been widely regarded as a light-weight and battery-less sensor finger or point-touch pen for manipulation,the touch screen for locating and sensing in recent years.In this paper,we only provides a 2D interaction with the devices,which may propose RF-Brush,a novel system based on linear tag array prevent the user from interacting with the computer via a attached onto linear shaped object,which can provide both the 3D manner.For example,people may intend to incline the 2D movement and the 3D orientation information of tagged point-touch pen to draw a bolder line,which is a common object.Here,the 2D movement refers to the moving trace manipulation for a real pen.But based on a traditional point-in 2D operating plane,whereas the 3D orientation refers to touch pen,people need to separately configure for the thick- the relative orientation of the object to the operating plane. ness of lines,which makes the manipulation tedious.Some The basic idea is to leverage the phase change of the linear dedicated devices,e.g.,Microsoft Surface Pen,can leverage tag array during the process of the 3D motion to recover the the embedded sensors to sense the 3D manipulations,such 2D movement in 2D operating plane and the 3D orientation as inclining the pen to get bolder handwriting,but they are in 3D environment.Particularly,we first extract the phase usually expensive and only work on specific smart devices.difference between different tags at the same time point,which Existing HCI approaches can be divided into four main can be used for the 3D orientation estimation via our proposed groups:computer vision-based,sensor-based,wifi-based and geometrical model.Moreover,we build another geometrical RFID-based.Computer vision-based approaches [1]-[3]and model,which can accurately track the 2D movement of the wireless techniques [4].[5],e.g.,WiFi or FMCW usually linear shaped object's tip based on the phase variation betweenRF-Brush: 3D Human-Computer Interaction via Linear Tag Array Yinyin Gong, Lei Xie, Chuyu Wang, Yanling Bu, Sanglu Lu State Key Laboratory for Novel Software Technology, Nanjing University Email: {yygong,wangcyu217}@dislab.nju.edu.cn, {lxie,sanglu}@nju.edu.cn, yanling@smail.nju.edu.cn Abstract—Nowadays, novel approaches of 3D human-computer interaction have enabled the capability of manipulating in the 3D space rather than 2D space. For example, Microsoft Surface Pen leverages the embedded sensors to sense the 3D manipulations, such as inclining the pen to get bolder handwriting. In this paper, we propose RF-Brush, a battery-free and light-weight solution for 3D human-computer interaction based on RFID, by simply attaching a linear RFID tag array onto the linear shaped object like a brush. RF-Brush senses the 3D orientation and 2D movement of the linear shaped object, when the human subject is drawing with this object in the 3D space. Here, the 3D orientation refers to the relative orientation of the linear shaped object to the operating plane, whereas the 2D movement refers to the moving trace in the 2D operating plane. In this way, we are able to transform an ordinary linear shaped object like a brush or pen to an intelligent HCI device. Particularly, we build two geometric models to depict the relationship between the RF-signal and the 3D orientation as well as 2D movement, respectively. Based on the geometric model, we propose the linear tag array-based HCI solution, implemented a prototype system, and evaluated the performance in real environment. The experiments show that RF-Brush achieves an average error of 5.7 ◦ and 8.6 ◦ of elevation and azimuthal angle, respectively, and an average error of 3.8cm and 4.2cm in movement tracking along X-axis and Y￾axis, respectively. Moreover, RF-Brush achieves 89% in letter recognition accuracy. I. INTRODUCTION With the ubiquity of sensing techniques, human-computer interaction (HCI) has drawn more and more attention recently, and novel approaches are required to provide a more natural interaction. Currently, due to the wide utilization of the smart devices (e.g., smart phone), the touch screen has become the most popular HCI interaction approach. However, by using the finger or point-touch pen for manipulation, the touch screen only provides a 2D interaction with the devices, which may prevent the user from interacting with the computer via a 3D manner. For example, people may intend to incline the point-touch pen to draw a bolder line, which is a common manipulation for a real pen. But based on a traditional point￾touch pen, people need to separately configure for the thick￾ness of lines, which makes the manipulation tedious. Some dedicated devices, e.g., Microsoft Surface Pen, can leverage the embedded sensors to sense the 3D manipulations, such as inclining the pen to get bolder handwriting, but they are usually expensive and only work on specific smart devices. Existing HCI approaches can be divided into four main groups:computer vision-based, sensor-based, wifi-based and RFID-based. Computer vision-based approaches [1]–[3] and wireless techniques [4], [5], e.g., WiFi or FMCW usually Z X 𝜷𝒂 Tag Array Y Operating Plane 𝜷𝒂 2D Movement 3D Orientation Fig. 1: Illustration of applications scenario of RF-Brush. need extensive training process. They either are sensitive to the change of background and light condition, or only pro￾vide coarse-grained classification manipulation. Sensor-based approaches [6] usually can accurately track the motion of the device in the 3D space based on the embedded IMU. But they usually suffer from the limited battery life. As for RFID-based approaches [7], [8], most of them are focusing on interacting with computer in 2D space. Therefore, according to the above considerations, in this paper, we aim to provide a battery-free and more natural 3D human-computer interaction by tracking the fine-grained 3D motion based on cheap ubiquitous objects. As Fig.1 shows, we interact with computer not only by the moving trace in 2D space, but also the orientation. Here, the orientation is able to reflect the thickness of lines, and the 2D moving trace reflects what we write finally. With the development of RFID technique, RFID tag has been widely regarded as a light-weight and battery-less sensor for locating and sensing in recent years. In this paper, we propose RF-Brush, a novel system based on linear tag array attached onto linear shaped object, which can provide both the 2D movement and the 3D orientation information of tagged object. Here, the 2D movement refers to the moving trace in 2D operating plane, whereas the 3D orientation refers to the relative orientation of the object to the operating plane. The basic idea is to leverage the phase change of the linear tag array during the process of the 3D motion to recover the 2D movement in 2D operating plane and the 3D orientation in 3D environment. Particularly, we first extract the phase difference between different tags at the same time point, which can be used for the 3D orientation estimation via our proposed geometrical model. Moreover, we build another geometrical model, which can accurately track the 2D movement of the linear shaped object’s tip based on the phase variation between