Depth Aware Finger Tapping on Virtual Displays MobiSys'18,June 10-15,2018,Munich,Germany Office with background music Hello wth speech noise :16 degree AUDIO PROCESS WDEO PROCESS Users Users (a)DolphinBoard user interface (b)FNR of different use cases (c)FNR of different environments Users Users (d)Input speed for different systems (e)FNR of different tapping angles (f)User experience Figure 14:DolphinBoard:In-the-air text input evaluation input text under three different use cases.The first use case is The text input speed of DolphinBoard is 12.18(SD=0.85)WPM to hold the smartphone in hand and type behind the phone.The and 13.1 (SD=1.2)WPM for single-finger and multi-finger inputs, second use case is to fix the smartphone on a selfie stick so that respectively.The text input speed was evaluated using the metrics the phone is more stable than in the first use case.The third use of Words Per Minute (WPM),which is defined as the number of case is to put the smartphone in a cardboard VR set worn on the 5-character words that the user can correctly enter for a duration head of the user as shown in Figure 1(b).Each of the users performs of one minute.As a reference,the typing speed on DolphinBoard is 500 taps under the three different user cases.As shown in Figure about two times that on Hololens,on which the users achieve about 14(b),the average FNR for tapping detection are 1.75%,1.23%,and 6.45 WPM on average as shown in Figure 14(d).The single-finger 2.03%,respectively.This shows that DolphinBoard is robust under input speed on DolphinBoard is limited by the finger tapping speed small interfering movements of the hands and head.The hand/head When we ask the users to tap continuously on the same key,the movement interferences in use case 1 and 3 only introduce a small average tapping speed is about 98 taps per minute,which converts increase in the FNR for less than 0.8%compared to the case that to about 19.6 WPM.While our method supports two-hand tracking. the device is fixed. users can only input text with a single hand due to the limited DolphinBoard is robust to noises and achieves low FNR in three viewing angle of mobile devices.As a result,multi-finger input different noisy environments.To evaluate the robustness of Dol- does not significantly increase the typing speed.The layout of the phinBoard,we asked users to perform finger tapping in three differ- QWERTY keyboard used in the current design of DolphinBoard ent noisy environments,including a cafe with 60dB speech noise, could also be a limitation of the text input speed.The users still need an office with 65dB background music,and playing 65dB music to move their hands during typing.which limits the speed.With from the same speaker that is used for playing the ultrasound.In all a better design of a dynamic virtual keyboard,the typing speed of these three environments,there are other people walking around. of DolphinBoard could be further increased.It is also possible to As shown in Figure 14(c),the average FNR for finger tapping detec- use the depth information provided by DolphinBoard to activate tion in the three different environments are 1.1%,1.35%,and 2.48%, different virtual keys.As shown in Figure 14(e),the average FNR respectively.Note that the ultrasound signals can be mixed with for DolphinBoard to detect gentle tappings and deep tappings are other audible signals.Our system can support the use case that 0.5%and 1.93%,respectively.Therefore,DolphinBoard can reliably users are using the speaker to play music while performing tapping detect different types of tappings and use this information to build detection using the ultrasound.When music is played on the same better keyboard layouts.For example,the gentle finger tappings speaker used by DolphinBoard,the intensity of the ultrasound is could be used for inputting the lower-case letters and the deep actually reduced due to the contention of the dynamical range on finger tappings could be used for inputting the capitalized letters. the speaker.However,DolphinBoard still achieved a low FNR of User experience evaluation:We evaluated the user experi- 2.48%in this challenging scenario. ence using 10 participants via questionnaire surveys,including 1)Depth Aware Finger Tapping on Virtual Displays MobiSys’18, June 10–15, 2018, Munich, Germany (a) DolphinBoard user interface 12345678 Users 0 0.5 1 1.5 2 2.5 False negative rate (%) Fix by selfie stick Hold in hand Set on the head (b) FNR of different use cases 12345678 Users 0 0.5 1 1.5 2 2.5 3 3.5 False negative rate (%) Office with background music Cafe with speech noise Music from the same speaker (c) FNR of different environments 12345678 Users 0 4 8 12 16 Words Per Minute (words/min) Single-finger Multi-finger Hololens (d) Input speed for different systems 12345678 Users 0 0.5 1 1.5 2 2.5 False negative rate (%) Deep tapping Gentle tapping (e) FNR of different tapping angles Technical complexity Accuracy Latency User friendliness User experience score 0 1 2 3 4 5 DolphinBoard Hololens (f) User experience Figure 14: DolphinBoard: In-the-air text input evaluation input text under three different use cases. The first use case is to hold the smartphone in hand and type behind the phone. The second use case is to fix the smartphone on a selfie stick so that the phone is more stable than in the first use case. The third use case is to put the smartphone in a cardboard VR set worn on the head of the user as shown in Figure 1(b). Each of the users performs 500 taps under the three different user cases. As shown in Figure 14(b), the average FNR for tapping detection are 1.75%, 1.23%, and 2.03%, respectively. This shows that DolphinBoard is robust under small interfering movements of the hands and head. The hand/head movement interferences in use case 1 and 3 only introduce a small increase in the FNR for less than 0.8% compared to the case that the device is fixed. DolphinBoard is robust to noises and achieves low FNR in three different noisy environments. To evaluate the robustness of Dol￾phinBoard, we asked users to perform finger tapping in three differ￾ent noisy environments, including a cafe with 60dB speech noise, an office with 65dB background music, and playing 65dB music from the same speaker that is used for playing the ultrasound. In all of these three environments, there are other people walking around. As shown in Figure 14(c), the average FNR for finger tapping detec￾tion in the three different environments are 1.1%, 1.35%, and 2.48%, respectively. Note that the ultrasound signals can be mixed with other audible signals. Our system can support the use case that users are using the speaker to play music while performing tapping detection using the ultrasound. When music is played on the same speaker used by DolphinBoard, the intensity of the ultrasound is actually reduced due to the contention of the dynamical range on the speaker. However, DolphinBoard still achieved a low FNR of 2.48% in this challenging scenario. The text input speed of DolphinBoard is 12.18 (SD=0.85) WPM and 13.1 (SD=1.2) WPM for single-finger and multi-finger inputs, respectively. The text input speed was evaluated using the metrics of Words Per Minute (WPM), which is defined as the number of 5-character words that the user can correctly enter for a duration of one minute. As a reference, the typing speed on DolphinBoard is about two times that on Hololens, on which the users achieve about 6.45 WPM on average as shown in Figure 14(d). The single-finger input speed on DolphinBoard is limited by the finger tapping speed. When we ask the users to tap continuously on the same key, the average tapping speed is about 98 taps per minute, which converts to about 19.6 WPM. While our method supports two-hand tracking, users can only input text with a single hand due to the limited viewing angle of mobile devices. As a result, multi-finger input does not significantly increase the typing speed. The layout of the QWERTY keyboard used in the current design of DolphinBoard could also be a limitation of the text input speed. The users still need to move their hands during typing, which limits the speed. With a better design of a dynamic virtual keyboard, the typing speed of DolphinBoard could be further increased. It is also possible to use the depth information provided by DolphinBoard to activate different virtual keys. As shown in Figure 14(e), the average FNR for DolphinBoard to detect gentle tappings and deep tappings are 0.5% and 1.93%, respectively. Therefore, DolphinBoard can reliably detect different types of tappings and use this information to build better keyboard layouts. For example, the gentle finger tappings could be used for inputting the lower-case letters and the deep finger tappings could be used for inputting the capitalized letters. User experience evaluation: We evaluated the user experi￾ence using 10 participants via questionnaire surveys, including 1)