AirTyping:A Mid-Air Typing Scheme based on Leap Motion UbiComp/ISWC'20 Adjunct,September 12-16,2020.Virtual Event,Mexico 012301230123 10P q0123 t h a t t h a t t h a t th a t 473473473 M p012 012 012 012 (a)Four Permutations for"that". 9 012 01 01 01 t h t o t o 475 47'3473473 01 2 012012 012 (b)One Permutation for "the".(c)Three Permutations for "to" Figure 3:The standard fingering for typing. Figure4:The permutations for“that",“the”and“to keystroke sequence Ki,P(KilWi)is the likelihood function which we further use the permutations of keystrokes to calculate the estimates the probability of the keystroke sequence Ki based on likelihood Pm(KilWi)for the keystroke sequence Ki based on the the word Wi.Since each word has same occurrence frequency, word Wi. i.e.,P(Wi)is equal among all words,and P(Ki)is also same for Specifically,for a detected keystroke sequence Ki =(k1,k2, all words,the Eg.(1)can be transformed to the calculation of the ....kn)and a word Wi (w1.w2.....wi,)in the dictionary,kp likelihood function P(KilWi).i.e.,P(WilKi)o P(KiWi),which will is the pth keystroke,and it is represented with the finger f(kp) be calculated with the number and the permutations of keystrokes. making the keystroke,p e [1,ni],f(kp)E [1,10].while wa is the gth letter of the word,wa e['a','z'],ge [1,li].If ni li,the 3.3.1 Inference with the number of keystrokes.Intuitively,if a key- number of the permutations of keystrokes in Ki for the word W stroke sequence Ki exactly matches with a word Wi,the number of keystrokes in Ki should be same with the number of chars in is A)=which denotes all possible cases of replacing Wi.Therefore,if the number of keystrokes is closer to the number ni letters in the word Wi with the ni keystrokes.If ni li,the of chars in a word,the word has a higher likelihood.For example number of permutations changes to A(n,l),which denotes all when four keystrokes are detected,the word "they"(i.e.,4 chars) cases of replacing the li keystrokes in Ki with lj letters in the word will have a higher likelihood than the word"a"(ie.,1 char)or"there Wi.Therefore,the likelihood Pm(KilW;)can be represented with fore"(i.e.,9 chars).Specifically,we use the number of keystrokes Eq.(4), ni in Ki to calculate the likelihood Pn(KilWj)for Ki based on the word Wi,as described in Eq.(2). ,、Πpetn P(wg∈Sfkp)lm),ni>lj Pm(AlWi)= meA(nt,）q∈L月 Pn(nilWj)=1-Imax +8 ni-lil (2) II pen P(wg∈Sfkp)lm,n:≤ meA(lj.n) Here,Ii is the length of word Wi,and Imax is the maximum length (4) difference of the words in the dictionary,i.e.,the difference between where m denotes a case of all permutations A.Given the permu- the longest and shortest word length.According to the adopted tation m,when the letter wa is in the key set Sf(k)typed by the word set [8].we set lmax=16 in this paper.To avoid the probability finger f(p)P(w()is calculated as For xam Pn(nilWi)being set to 0,we introduce a tolerance factor =0.01. ple,the letter't'is in the key set of the finger 4,so it is calculated as For a better illustration,we assume four keystrokes are detected. If the letterw is not in the key set S()the probability and we calculate the likelihood of the words“they”,“a"and“"there. P(wae Sf(kp)lm)is set to the tolerance factor 8. fore"with Eq.(2).As shown in Eq.(3),the likelihood of the word For instance,if the user wants to type the word"the",the 4th.7th "they”is higher than that of the words"a”or“therefore”,since the and 3rd finger will press in sequence.Based on Fig.3,the detected number (i.e.,four)of chars in "they"is closer to the number (i.e., keystroke sequence Ki will be represented as'4'-'7'-'3,and the four)of keystrokes. number of Ki is ni=3.In regard to the word Wi,if Wi="that",the 4-1川 Pn(ni=4lW=“a"=1- =0.813 length of Wi is lj =4.Since ni<li,the number of permutations is 16+0.01 A(lj,ni).The corresponding possible permutations are shown in Pn(ni=4|w=“they"）)=1- 4-4到 Fig.4(a).Then the likelihood of the keystroke sequence Ki based =1 (3) 16+0.01 on the word Wi can be calculated with Eq.(5). 14-9 Pn(ni =4Wi="therefore")=1- =0.688 1 1 1 16+0.01 Pma,ni训w=hat)=商×高xi+ 3.3.2 Inference with the Permutations of Keystrokes.In addition to (5) 1 the number of keystrokes,the finger sequence making the keystroke 阿×6+5×02+6=575×10-4 sequence also affects the likelihood,since each finger maps with more than one character.As shown in Fig.3,when the user types where is the probability of typing't'by finger 4,and is the in standard fingering,each finger makes several fixed keystrokes, probability of typing 'h'by finger 7.The four components represent i.e.,the keys have the same color with the fingertip.Therefore, the four permutation cases shown in Fig.4(a).AirTyping: A Mid-Air Typing Scheme based on Leap Motion UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico 1 2 3 4 5 6 7 89 10 Q W E R TYU I O P Z XCVB N M A S D F GHJK L space Figure 3: The standard fingering for typing. keystroke sequence Ki , P (Ki |Wj ) is the likelihood function which estimates the probability of the keystroke sequence Ki based on the word Wj . Since each word has same occurrence frequency, i.e., P (Wj ) is equal among all words, and P (Ki ) is also same for all words, the Eq. (1) can be transformed to the calculation of the likelihood function P (Ki |Wj ), i.e., P (Wj |Ki ) ∝ P (Ki |Wj ), which will be calculated with the number and the permutations of keystrokes. 3.3.1 Inference with the number of keystrokes. Intuitively, if a key￾stroke sequence Ki exactly matches with a word Wj , the number of keystrokes in Ki should be same with the number of chars in Wj . Therefore, if the number of keystrokes is closer to the number of chars in a word, the word has a higher likelihood. For example, when four keystrokes are detected, the word “they” (i.e., 4 chars) will have a higher likelihood than the word “a” (i.e., 1 char) or “there￾fore” (i.e., 9 chars). Specifically, we use the number of keystrokes ni in Ki to calculate the likelihood Pn (Ki |Wj ) for Ki based on the word Wj , as described in Eq. (2). Pn (ni |Wj ) = 1 − |ni − lj | lmax + δ (2) Here, lj is the length of word Wj , and lmax is the maximum length difference of the words in the dictionary, i.e., the difference between the longest and shortest word length. According to the adopted word set [8], we setlmax = 16 in this paper. To avoid the probability Pn (ni |Wj ) being set to 0, we introduce a tolerance factor δ = 0.01. For a better illustration, we assume four keystrokes are detected, and we calculate the likelihood of the words “they”, “a” and “there￾fore” with Eq. (2). As shown in Eq. (3), the likelihood of the word “they” is higher than that of the words “a” or “therefore”, since the number (i.e., four) of chars in “they” is closer to the number (i.e., four) of keystrokes. Pn (ni = 4|Wj = “a”) = 1 − |4 − 1| 16 + 0.01 = 0.813 Pn (ni = 4|Wj = “they”) = 1 − |4 − 4| 16 + 0.01 = 1 Pn (ni = 4|Wj = “therefore”) = 1 − |4 − 9| 16 + 0.01 = 0.688 (3) 3.3.2 Inference with the Permutations of Keystrokes. In addition to the number of keystrokes, the finger sequence making the keystroke sequence also affects the likelihood, since each finger maps with more than one character. As shown in Fig. 3, when the user types in standard fingering, each finger makes several fixed keystrokes, i.e., the keys have the same color with the fingertip. Therefore, t h a t q 0 1 2 3 p 0 1 2 0 1 2 0 1 2 0 1 2 '4' '7' '3' 0 1 2 3 0 1 2 3 0 1 2 3 (a) Four Permutations for "that". q 0 1 2 0 1 0 1 0 1 t h e t o (b) One Permutation for "the". (c) Three Permutations for “to". p 0 1 2 0 1 2 0 1 2 0 1 2 '4' '7' '3' '4' '7' '3' '4' '7' '3' '4' '7' '3' '4' '7' '3' '4' '7' '3' '4' '7' '3' t h a t t h a t t h a t t o t o Figure 4: The permutations for “that”, “the” and “to”. we further use the permutations of keystrokes to calculate the likelihood Pm (Ki |Wj ) for the keystroke sequence Ki based on the word Wj . Specifically, for a detected keystroke sequence Ki = (k1, k2, ..., kni ) and a word Wj = (w1,w2, ...,wlj ) in the dictionary, kp is the pth keystroke, and it is represented with the finger f (kp ) making the keystroke, p ∈ [1,ni], f (kp ) ∈ [1, 10], while wq is the qth letter of the word, wq ∈[‘a’, ‘z’], q ∈ [1,lj]. If ni ≤ lj , the number of the permutations of keystrokes in Ki for the word Wj is A(lj ,ni ) = lj ! (lj−ni )! , which denotes all possible cases of replacing ni letters in the word Wj with the ni keystrokes. If ni > lj , the number of permutations changes to A(ni,lj ), which denotes all cases of replacing the lj keystrokes in Ki with lj letters in the word Wj . Therefore, the likelihood Pm (Ki |Wj ) can be represented with Eq. (4), Pm (A|Wj ) = ⎧⎪⎪⎪⎪ ⎨ ⎪⎪⎪⎪ ⎩ m∈A(ni,lj ) p ∈ [1, ni ] q ∈ [1, lj ] P (wq ∈ Sf (kp ) |m),ni > lj m∈A(lj,ni ) p ∈ [1, ni ] q ∈ [1, lj ] P (wq ∈ Sf (kp ) |m),ni ≤ lj (4) where m denotes a case of all permutations A. Given the permu￾tation m, when the letter wq is in the key set Sf (kp ) typed by the finger f (kp ), P (wq ∈ Sf (kp ) |m) is calculated as 1 |Sf (kp ) | . For exam￾ple, the letter ‘t’ is in the key set of the finger 4, so it is calculated as 1 |S4 | = 1 6 . If the letter wq is not in the key set Sf (kp ), the probability P (wq ∈ Sf (kp ) |m) is set to the tolerance factor δ. For instance, if the user wants to type the word “the”, the 4th, 7th and 3rd finger will press in sequence. Based on Fig. 3, the detected keystroke sequence Ki will be represented as ‘4’-‘7’-‘3’, and the number of Ki is ni = 3. In regard to the wordWj , ifWj = “that”, the length of Wj is lj = 4. Since ni < lj , the number of permutations is A(lj ,ni ). The corresponding possible permutations are shown in Fig. 4(a). Then the likelihood of the keystroke sequence Ki based on the word Wj can be calculated with Eq. (5). Pm (A(lj ,ni )|Wj = “that”) = 1 |S4 | × 1 |S7 | × δ + 1 |S4 | × 1 |S7 | × δ + 1 |S4 | × δ 2 + δ 3 = 5.75 × 10−4 (5) where 1 |S4 | is the probability of typing ‘t’ by finger 4, and 1 |S7 | is the probability of typing ‘h’ by finger 7. The four components represent the four permutation cases shown in Fig. 4(a).