MobiSys'18,June 10-15,2018,Munich,Germany Ke Sun et al. Users Users (a)DolphinPiano user interface (b)Tempo for DolphinPiano (c)TPR of different feedback (d)User experience Figure 15:DolphinPiano:AR piano evaluation technical complexity,2)accuracy.3)latency and 4)user friendli- DolphinPiano provides accurate feedback for the bending angle ness(5=strongly positive,1=strongly negative).Users were asked of fingers.To evaluate the accuracy of bending angle feedback,we to type using DolphinBoard and Hololens for 20 minutes per per- asked the participants to play the piano with different bending son.Figure 14(f)shows the results.For the technical complexity, angles.Four levels of bending angles were used in the experiement, some participants have some negative evaluation,this is mainly i.e,0~20,20 ~40,40 ~60,60~90 degrees.Participants did not because DolphinBoard only needs user to move their finger to the know the ground truth of their bending angles,they could only expected location instead of moving their head like Hololens.For adjust their movement by the feedback of DolphinPiano.As shown the accuracy,most of the participants hold positive/strongly posi- in 15(c),DolphinPiano achieved an average accuracy of 95.4%,93.2%. tive attitudes because DolphinBoard is able to detect small finger 93.8%,and 97.4%TPR for different bending angles,respectively. tappings.For the latency,most of the people have positive/strongly Participants had a good grasp of bending angle feedback after a positive evaluation since ultrasound-assisted methods shorten the short training period. detecting duration effectively.For the user friendliness,the tapping- User experience evaluation:We evaluated the user experi- in-the-air mechanisms are acceptable for most of users.In addition, ence of eight participants via the questionnaire surveys,including most participants(9/10)are in favor of the idea of different tapping 1)feedback accuracy and 2)angle sensitivity(5=strongly positive. depths to input lower and upper case letters 1=strongly negative).Figure 15(d)shows the results.All the par- 8.5.2 DolphinPiano:AR piano based on the finger bending angle. ticipants were positive to bending angle feedback.With the help This study aims at providing feedback based on different bending of bending angle feedback,they could play just like on a physical angles of finger tappings.In this case study,we develop Dolphin keyboard.For bending angle sensitivity,they were surprised that Piano,which enables users to play the piano in-the-air.For physical DolphinPiano could measure the bending angle by using one cam- pianos,the loudness of the notes depends on the strength of the era on the mobile devices.Although there was no tactile feedback, keystroke:the greater the velocity of a keystroke,the greater the they could play smoothly with the help of different levels of audio force of the hammer hitting the strings,and the louder the sound of and visual feedbacks. the note produced.Since our system is able to measure the micro movement of the finger,DolphinPiano simulates different levels of 9 CONCLUSIONS force being applied to keys based on the bending angle. In this paper,we make three key contributions.First,we propose User interface:As shown in Figure 15(a),a piano keyboard with combining ultrasound sensing information and vision information one octave is rendered on the screen.This keyboard is translucent so to achieve fine-grained bending angle measurements for in-air that users can locate their fingertips to the virtual key on the screen. finger tappings.Second,we design a tapping-in-the-air scheme White keys and black keys have size of 150 x 640 and 100 x 320 with high accuracy and low latency.Third,we implemented and in pixels,respectively.When users press the key,the application evaluated our system using commercial smartphones without any plays the corresponding note with different volume based on the hardware modification.Our experimental results show that our finger bending angle. system achieves high accuracy,low latency and provides feedback Testing Participants:Eight people participated in case study 1 based on different bending angles of finger tappings.One limitation were also invited to use the DolphinPiano application.Since they of our solution is that its audio system incurs considerable overhead were familiar with DolphinBoard,they were asked to play a simple in power consumption.One possible solution for this is to design piece of music (Jingle Bells)without any training.Each person uses low-power speakers that are specialized for emitting ultrasounds. our DolphinPiano application for 20 minutes. Performance evaluation:The tempo of DolphinPiano achieves ACKNOWLEDGMENTS 75.0(SD=3.88)BPM (beats per minute)and 79.3(SD=5.31)BPM for We would like to thank our shepherd Yuvraj Agarwal and anony- single-finger and multi-finger inputs,respectively.To evaluate the mous reviewers for their valuable comments.This work is partially music tempo,eight users were asked to play music on Dolphin- supported by National Natural Science Foundation of China under Piano application as fast as possible.As shown in Figure 15(b). Grant Numbers61472185,61472184,61373129,61502229,61672353, tapping on DolphinPiano is faster than on DolphinBoard,this is and 61472252.the National Science Foundation under Grant Num- mainly because that the keys in DolphinPiano are larger than those bers CNS-1421407,Collaborative Innovation Center of Novel Soft in text input which means shorter movement time according to ware Technology and Industrialization,and the Jiangsu High-level Fitts'Law [19] Innovation and Entrepreneurship(Shuangchuang)Program.MobiSys’18, June 10–15, 2018, Munich, Germany Ke Sun et al. (a) DolphinPiano user interface 12345678 Users 60 65 70 75 80 85 90 95 Beats Per Minute (beats/min) Single-finger Multi-finger (b) Tempo for DolphinPiano 12345678 Users 85 87 89 91 93 95 97 99 True Positive Rate (%) 0~20o 20~40o 40~60o 60~90o (c) TPR of different feedback 12345678 Users 1 2 3 4 5 User experience score Bending angles feedback Bending angles sensitivity (d) User experience Figure 15: DolphinPiano: AR piano evaluation technical complexity, 2) accuracy, 3) latency and 4) user friendli￾ness (5=strongly positive, 1=strongly negative). Users were asked to type using DolphinBoard and Hololens for 20 minutes per per￾son. Figure 14(f) shows the results. For the technical complexity, some participants have some negative evaluation, this is mainly because DolphinBoard only needs user to move their finger to the expected location instead of moving their head like Hololens. For the accuracy, most of the participants hold positive/strongly posi￾tive attitudes because DolphinBoard is able to detect small finger tappings. For the latency, most of the people have positive/strongly positive evaluation since ultrasound-assisted methods shorten the detecting duration effectively. For the user friendliness, the tapping￾in-the-air mechanisms are acceptable for most of users. In addition, most participants (9/10) are in favor of the idea of different tapping depths to input lower and upper case letters. 8.5.2 DolphinPiano: AR piano based on the finger bending angle. This study aims at providing feedback based on different bending angles of finger tappings. In this case study, we develop Dolphin￾Piano, which enables users to play the piano in-the-air. For physical pianos, the loudness of the notes depends on the strength of the keystroke: the greater the velocity of a keystroke, the greater the force of the hammer hitting the strings, and the louder the sound of the note produced. Since our system is able to measure the micro movement of the finger, DolphinPiano simulates different levels of force being applied to keys based on the bending angle. User interface: As shown in Figure 15(a), a piano keyboard with one octave is rendered on the screen. This keyboard is translucent so that users can locate their fingertips to the virtual key on the screen. White keys and black keys have size of 150 × 640 and 100 × 320 in pixels, respectively. When users press the key, the application plays the corresponding note with different volume based on the finger bending angle. Testing Participants: Eight people participated in case study 1 were also invited to use the DolphinPiano application. Since they were familiar with DolphinBoard, they were asked to play a simple piece of music (Jingle Bells) without any training. Each person uses our DolphinPiano application for 20 minutes. Performance evaluation: The tempo of DolphinPiano achieves 75.0 (SD=3.88) BPM (beats per minute) and 79.3 (SD=5.31) BPM for single-finger and multi-finger inputs, respectively. To evaluate the music tempo, eight users were asked to play music on Dolphin￾Piano application as fast as possible. As shown in Figure 15(b), tapping on DolphinPiano is faster than on DolphinBoard, this is mainly because that the keys in DolphinPiano are larger than those in text input which means shorter movement time according to Fitts’ Law [19]. DolphinPiano provides accurate feedback for the bending angle of fingers. To evaluate the accuracy of bending angle feedback, we asked the participants to play the piano with different bending angles. Four levels of bending angles were used in the experiement, i.e., 0 ∼ 20, 20 ∼ 40, 40 ∼ 60, 60 ∼ 90 degrees. Participants did not know the ground truth of their bending angles, they could only adjust their movement by the feedback of DolphinPiano. As shown in 15(c), DolphinPiano achieved an average accuracy of 95.4%, 93.2%, 93.8%, and 97.4% TPR for different bending angles, respectively. Participants had a good grasp of bending angle feedback after a short training period. User experience evaluation: We evaluated the user experi￾ence of eight participants via the questionnaire surveys, including 1) feedback accuracy and 2) angle sensitivity (5=strongly positive, 1=strongly negative). Figure 15(d) shows the results. All the par￾ticipants were positive to bending angle feedback. With the help of bending angle feedback, they could play just like on a physical keyboard. For bending angle sensitivity, they were surprised that DolphinPiano could measure the bending angle by using one cam￾era on the mobile devices. Although there was no tactile feedback, they could play smoothly with the help of different levels of audio and visual feedbacks. 9 CONCLUSIONS In this paper, we make three key contributions. First, we propose combining ultrasound sensing information and vision information to achieve fine-grained bending angle measurements for in-air finger tappings. Second, we design a tapping-in-the-air scheme with high accuracy and low latency. Third, we implemented and evaluated our system using commercial smartphones without any hardware modification. Our experimental results show that our system achieves high accuracy, low latency and provides feedback based on different bending angles of finger tappings. One limitation of our solution is that its audio system incurs considerable overhead in power consumption. One possible solution for this is to design low-power speakers that are specialized for emitting ultrasounds. ACKNOWLEDGMENTS We would like to thank our shepherd Yuvraj Agarwal and anony￾mous reviewers for their valuable comments. This work is partially supported by National Natural Science Foundation of China under Grant Numbers 61472185, 61472184, 61373129, 61502229, 61672353, and 61472252, the National Science Foundation under Grant Num￾bers CNS-1421407, Collaborative Innovation Center of Novel Soft￾ware Technology and Industrialization, and the Jiangsu High-level Innovation and Entrepreneurship (Shuangchuang) Program