VSkin:Sensing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals Ke Sun Ting Zhao State Key Laboratory for Novel Software Technology State Key Laboratory for Novel Software Technology Nanjing University,China,kesun@smail.nju.edu.cn Nanjing University,zhaoting@smail.nju.edu.cn Wei Wang Lei Xie State Key Laboratory for Novel Software Technology State Key Laboratory for Novel Software Technology Nanjing University,China,ww@nju.edu.cn Nanjing University,China,lxie@nju.edu.cn ABSTRACT Enabling touch gesture sensing on all surfaces of the mo- bile device,not limited to the touchscreen area,leads to new user interaction experiences.In this paper,we propose VSkin,a system that supports fine-grained gesture-sensing on the back of mobile devices based on acoustic signals. VSkin utilizes both the structure-borne sounds,i.e.,sounds propagating through the structure of the device,and the air-borne sounds,i.e.,sounds propagating through the air, (a)Back-Swiping (b)Back-Tapping (c)Back-Scrolling to sense finger tapping and movements.By measuring both Figure 1:Back-of-Device interactions the amplitude and the phase of each path of sound signals, VSkin detects tapping events with an accuracy of 99.65%and captures finger movements with an accuracy of 3.59 mm. 1 INTRODUCTION Touch gesture is one of the most important ways for users CCS CONCEPTS to interact with mobile devices.With the wide-deployment of touchscreens,a set of user-friendly touch gestures,such Human-centered computing-Interface design as swiping,tapping,and scrolling,have become the de facto prototyping;Gestural input; standard user interface for mobile devices.However,due to KEYWORDS the high-cost of the touchscreen hardware,gesture-sensing is usually limited to the front surface of the device.Further- Touch gestures;Ultrasound more,touchscreens combine the function of gesture-sensing ACM Reference Format: with the function of displaying.This leads to the occlusion Ke Sun,Ting Zhao,Wei Wang,and Lei Xie.2018.VSkin:Sens- problem [30],i.e.,user fingers often block the content dis- ing Touch Gestures on Surfaces of Mobile Devices Using Acoustic played on the screen during the interaction process Signals.In MobiCom'18:24th Annual International Conference on Enabling gesture-sensing on all surfaces of the mobile de- Mobile Computing and Networking,October 29-November 2,2018, vice,not limited to the touchscreen area,leads to new user New Delhi,India.ACM,New York,NY,USA,15 pages.https://doi. interaction experiences.First,new touch gestures solve the rg/10.1145/3241539.3241568 occlusion problem of the touchscreen.For example,Back-of- Device(BoD)gestures use tapping or swiping on the back of Permission to make digital or hard copies of all or part of this work for a smartphone as a supplementary input interface [22,35].As personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear shown in Figure 1,the screen is no longer blocked when the this notice and the full citation on the first page.Copyrights for components back-scrolling gesture is used for scrolling the content.BoD of this work owned by others than ACM must be honored.Abstracting with gestures also enrich the user experience of mobile games by credit is permitted.To copy otherwise,or republish,to post on servers or to allowing players to use the back surface as a touchpad.Sec- redistribute to lists,requires prior specific permission and/or a fee.Request ond,defining new touch gestures on different surfaces helps permissions from permissions@acm.org. the system better understand user intentions.On traditional MobiCom'18,October 29-November 2,2018,New Delhi,India e2018 Association for Computing Machinery. touchscreens,touching a webpage on the screen could mean ACM ISBN978-1-4503-5903-0/18/10..$15.00 that the user wishes to click a hyperlink or the user just https:/doi.org/10.1145/3241539.3241568 wants to scroll down the page.Existing touchscreen schemesVSkin: Sensing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals Ke Sun State Key Laboratory for Novel Software Technology Nanjing University, China, kesun@smail.nju.edu.cn Ting Zhao State Key Laboratory for Novel Software Technology Nanjing University, zhaoting@smail.nju.edu.cn Wei Wang State Key Laboratory for Novel Software Technology Nanjing University, China, ww@nju.edu.cn Lei Xie State Key Laboratory for Novel Software Technology Nanjing University, China, lxie@nju.edu.cn ABSTRACT Enabling touch gesture sensing on all surfaces of the mo￾bile device, not limited to the touchscreen area, leads to new user interaction experiences. In this paper, we propose VSkin, a system that supports fine-grained gesture-sensing on the back of mobile devices based on acoustic signals. VSkin utilizes both the structure-borne sounds, i.e., sounds propagating through the structure of the device, and the air-borne sounds, i.e., sounds propagating through the air, to sense finger tapping and movements. By measuring both the amplitude and the phase of each path of sound signals, VSkin detects tapping events with an accuracy of 99.65% and captures finger movements with an accuracy of 3.59 mm. CCS CONCEPTS • Human-centered computing → Interface design prototyping; Gestural input; KEYWORDS Touch gestures; Ultrasound ACM Reference Format: Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. VSkin: Sens￾ing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals. In MobiCom ’18: 24th Annual International Conference on Mobile Computing and Networking, October 29–November 2, 2018, New Delhi, India. ACM, New York, NY, USA, 15 pages. https://doi. org/10.1145/3241539.3241568 Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. MobiCom’18, October 29–November 2, 2018, New Delhi, India © 2018 Association for Computing Machinery. ACM ISBN 978-1-4503-5903-0/18/10. . . $15.00 https://doi.org/10.1145/3241539.3241568 (a) Back-Swiping (b) Back-Tapping (c) Back-Scrolling Figure 1: Back-of-Device interactions 1 INTRODUCTION Touch gesture is one of the most important ways for users to interact with mobile devices. With the wide-deployment of touchscreens, a set of user-friendly touch gestures, such as swiping, tapping, and scrolling, have become the de facto standard user interface for mobile devices. However, due to the high-cost of the touchscreen hardware, gesture-sensing is usually limited to the front surface of the device. Further￾more, touchscreens combine the function of gesture-sensing with the function of displaying. This leads to the occlusion problem [30], i.e., user fingers often block the content dis￾played on the screen during the interaction process. Enabling gesture-sensing on all surfaces of the mobile de￾vice, not limited to the touchscreen area, leads to new user interaction experiences. First, new touch gestures solve the occlusion problem of the touchscreen. For example, Back-of￾Device (BoD) gestures use tapping or swiping on the back of a smartphone as a supplementary input interface [22, 35]. As shown in Figure 1, the screen is no longer blocked when the back-scrolling gesture is used for scrolling the content. BoD gestures also enrich the user experience of mobile games by allowing players to use the back surface as a touchpad. Sec￾ond, defining new touch gestures on different surfaces helps the system better understand user intentions. On traditional touchscreens, touching a webpage on the screen could mean that the user wishes to click a hyperlink or the user just wants to scroll down the page. Existing touchscreen schemes