UbiComp/ISWC'20 Adjunct,September 12-16,2020,Virtual Event,Mexico Hao Zhang et al. 0.98 0m- 002 3 0as 4002 0.1 5 a92 Q. 6 .2 8 0. 0B2 9 三10 是db1om0e 0624 02 04a020202 2345678910ul Detected finger making a keystroke lop-4 top-7 top-12 top-27 Figure 5:The detection accuracy of the finger making a key- Figure 6:The probability that top-k candidate words contain stroke(null'means no keystrokes). the typed/target word. If the length of the word and that of the keystroke sequence that the top-1 candidate word with the highest likelihood is the are the same (i.e.,ni=li),there is only one permutation for them, typed/target word achieves 90.2%.Besides,the probability that as shown in Fig.4(b),and the likelihood about the word "the"is the top-3 candidate words contain the typed/target word achieves shown in Eq.(6).If ni>lj,we assume the word Wj="to".At this 98.5%.Overall,we can detect keystrokes and infer the typed words time,ni=3 and li=2,the number of permutations changes to accurately,and provide an efficient mid-air typing scheme for text A(ni,li)=3.The corresponding possible permutations are shown input in Fig.4(c).We will calculate the likelihood of K;based on the word Wi with Eq.(7): 5 CONCLUSION In this paper,we propose AirTyping,which allows people to type in Pm(a,nIw=he)=a×S7i×S mid-air based on Leap Motion.To detect the possible keystrokes,we (6) introduce the bending angles of fingers,movement trend of a finger =1.85×102 in consecutive coordinates,and time difference between keystrokes. 1 Pm(A(ni,li)Wi="to")= 1 商×+ ×6+62 To infer the typed word sequence,we introduce Bayesian method (7) and calculate the likelihood function from the number and the =4.33×10-3 permutation of keystrokes.The experiment results show that Air- Typing can detect the keystrokes and infer the typed text efficiently, 3.3.3 Combination of the number and the permutation of keystrokes. i.e.,the true positive rate of keystroke detection is 92.2%,while the Finally,we combine the probability about the number and the per- accuracy that the top-1 inferred word is the typed word achieves mutations of the keystroke sequence,and formulate it as P(KiWi)= 90.2%. Pn(nilWi)x Pm(AlWi).For a detected keystroke sequence Ki,we first filter out the words that satisfy Ini-lils An,and then calcu- ACKNOWLEDGMENTS late the likelihood for each word,and select the word having the This work is supported by National Natural Science Foundation of highest probability as the inferred result.Here,An =2. China under Grant Nos.61802169.61872174.61832008.61902175 61906085;JiangSu Natural Science Foundation under Grant Nos. 4 PERFORMANCE EVALUATION BK20180325,BK20190293;the Key R&D Program of Jiangsu Province We implement AirTyping based on the Leap Motion Controller under Grant No.BE2018116.This work is partially supported by (LMC),as shown in Fig.1.The LMC uses the embedded cameras Collaborative Innovation Center of Novel Software Technology and and infrared LEDs to provide the coordinates of finger joints,where Industrialization.Yafeng Yin is the corresponding author. the sampling rate is 60 Hz.The user performs typing behaviors about 15cm above LMC,while the inferred words will be sent to the REFERENCES displayer for text input.The adopted dictionary includes 5000 most [1]Microsoft Hololens,https://www.microsoft.com/en-us/hololens. [2]L Xie and C.Wang and A.X.Liu and J.Sun and S.Lu,Multi-Touch in the Air:Con- frequently used words downloaded from Word Frequency Data [8]. current Micromovement Recognition Using RF Signals,in IEEE/ACM Transactions Firstly,we evaluate the performance of keystroke detection mod- on Networking,2018. ule.Specifically,each finger makes 50 keystrokes.As shown in Fig. [3]Y.Yin and Q.Li and L.Xie and S.Yi and E.Novak and S.Lu.CamK:Camera- Based Keystroke Detection and Localization for Small Mobile Devices,in IEEE 5,the average detection accuracy of the finger making a keystroke Transactions on Mobile Computing,2018. reaches 92.2%,and the false positive rate (i.e.,treat non-keystrokes [4]X.Yi and C.Yu and M.Zhang and S.Gao and K.Sun and Y.Shi,ATK:Enabling ten- finger freehand typing in air based on 3d hand tracking data,in ACM Symposium as keystrokes)and false negative rate (i.e.,treat keystrokes as non- on User Interface 2015. keystrokes)are 1.7%and 5.4%,respectively.Thus we can accurately [5]A.Boudjelthia and S.Nasim and J.Eskola and J.Adeegbe and O.Hourula and S analyze the finger movements and detect the possible keystrokes. Klakegg and D.Ferreira,Enabling Mid-air Browser Interaction with Leap Motion in ACM International Symposium on Pervasive and Ubiquitous Computing and In addition,we test the performance of word inference module. Wearable Computers,2018. Specifically,we calculate the likelihood function for each word [6]Leap Motion,https://developer.leapmotion.co and obtain the candidate words in the dictionary.As shown in [7]An average professional typist types usually in speeds of 50 to 80 wpm,https: //en.wikipedia.org/wiki/Words-per-minute. Fig.6,when the keystrokes are detected accurately,the probability [8]Word Frequency Data Set,https://www.wordfrequency.infoUbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico Hao Zhang et al. 0.98 0.02 0.02 0.9 0.082 0.92 0.08 0.02 0.88 0.1 0.92 0.082 0.98 0.021 0.92 0.082 0.98 0.02 0.041 0.9 0.061 0.96 0.041 0.041 0.02 0.02 0.02 0.041 0.02 0.02 0.82 0 1 2 3 4 5 6 7 8 9 null Detected finger making a keystroke 0 1 2 3 4 5 6 7 8 9 Actual finger making a keystroke null 123456789

1 2 3 4 5 6 7 8 9 Figure 5: The detection accuracy of the finger making a key￾stroke (‘null’ means no keystrokes). If the length of the word and that of the keystroke sequence are the same (i.e., ni = lj), there is only one permutation for them, as shown in Fig. 4(b), and the likelihood about the word “the” is shown in Eq. (6). If ni > lj , we assume the word Wj = “to”. At this time, ni = 3 and lj = 2, the number of permutations changes to A(ni,lj ) = 3. The corresponding possible permutations are shown in Fig. 4(c). We will calculate the likelihood of Ki based on the word Wj with Eq. (7): Pm (A(lj ,ni )|Wj = “the”) = 1 |S4 | × 1 |S7 | × 1 |S3 | = 1.85 × 10−2 (6) Pm (A(ni,lj )|Wj = “to”) = 1 |S4 | × δ + 1 |S4 | × δ + δ 2 = 4.33 × 10−3 (7) 3.3.3 Combination of the number and the permutation of keystrokes. Finally, we combine the probability about the number and the per￾mutations of the keystroke sequence, and formulate it as P (Ki |Wj ) = Pn (ni |Wj ) × Pm (A|Wj ). For a detected keystroke sequence Ki , we first filter out the words that satisfy |ni − lj | ≤ Δn, and then calcu￾late the likelihood for each word, and select the word having the highest probability as the inferred result. Here, Δn = 2. 4 PERFORMANCE EVALUATION We implement AirTyping based on the Leap Motion Controller (LMC), as shown in Fig. 1. The LMC uses the embedded cameras and infrared LEDs to provide the coordinates of finger joints, where the sampling rate is 60 Hz. The user performs typing behaviors about 15cm above LMC, while the inferred words will be sent to the displayer for text input. The adopted dictionary includes 5000 most frequently used words downloaded from Word Frequency Data [8]. Firstly, we evaluate the performance of keystroke detection mod￾ule. Specifically, each finger makes 50 keystrokes. As shown in Fig. 5, the average detection accuracy of the finger making a keystroke reaches 92.2%, and the false positive rate (i.e., treat non-keystrokes as keystrokes) and false negative rate (i.e., treat keystrokes as non￾keystrokes) are 1.7% and 5.4%, respectively. Thus we can accurately analyze the finger movements and detect the possible keystrokes. In addition, we test the performance of word inference module. Specifically, we calculate the likelihood function for each word and obtain the candidate words in the dictionary. As shown in Fig. 6, when the keystrokes are detected accurately, the probability 90.16 96.56 98.48 99.3 99.62 99.9 99.96 99.98 100 top-1 top-2 top-3 top-4 top-5 top-6 top-7 top-12 top-27 Candidate words 0 20 40 60 80 100 Probability (%) Figure 6: The probability that top-k candidate words contain the typed/target word. that the top-1 candidate word with the highest likelihood is the typed/target word achieves 90.2%. Besides, the probability that the top-3 candidate words contain the typed/target word achieves 98.5%. Overall, we can detect keystrokes and infer the typed words accurately, and provide an efficient mid-air typing scheme for text input. 5 CONCLUSION In this paper, we propose AirTyping, which allows people to type in mid-air based on Leap Motion. To detect the possible keystrokes, we introduce the bending angles of fingers, movement trend of a finger in consecutive coordinates, and time difference between keystrokes. To infer the typed word sequence, we introduce Bayesian method and calculate the likelihood function from the number and the permutation of keystrokes. The experiment results show that Air￾Typing can detect the keystrokes and infer the typed text efficiently, i.e., the true positive rate of keystroke detection is 92.2%, while the accuracy that the top-1 inferred word is the typed word achieves 90.2%. ACKNOWLEDGMENTS This work is supported by National Natural Science Foundation of China under Grant Nos. 61802169, 61872174, 61832008, 61902175, 61906085; JiangSu Natural Science Foundation under Grant Nos. BK20180325, BK20190293; the Key R&D Program of Jiangsu Province under Grant No. BE2018116. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Yafeng Yin is the corresponding author. REFERENCES [1] Microsoft Hololens, https://www.microsoft.com/en-us/hololens. [2] L. Xie and C. Wang and A. X. Liu and J. Sun and S. Lu, Multi-Touch in the Air: Con￾current Micromovement Recognition Using RF Signals, in IEEE/ACM Transactions on Networking, 2018. [3] Y. Yin and Q. Li and L. Xie and S. Yi and E. Novak and S. Lu, CamK: Camera￾Based Keystroke Detection and Localization for Small Mobile Devices, in IEEE Transactions on Mobile Computing, 2018. [4] X. Yi and C. Yu and M. Zhang and S. Gao and K. Sun and Y. Shi, ATK: Enabling ten￾finger freehand typing in air based on 3d hand tracking data, in ACM Symposium on User Interface Software Technology, 2015. [5] A. Boudjelthia and S. Nasim and J. Eskola and J. Adeegbe and O. Hourula and S. Klakegg and D. Ferreira, Enabling Mid-air Browser Interaction with Leap Motion, in ACM International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers, 2018. [6] Leap Motion, https://developer.leapmotion.com. [7] An average professional typist types usually in speeds of 50 to 80 wpm, https: //en.wikipedia.org/wiki/Words-per-minute. [8] Word Frequency Data Set, https://www.wordfrequency.info.