UbiComp/ISWC'20 Adjunct,September 12-16,2020,Virtual Event,Mexico Hao Zhang et al. keystrokes,and the one-to-many mapping between the finger and of a finger.For convenience,we introduce 0 to describe the bending characters in standard fingering,we introduce the Bayesian method angle,which is formed by the distal phalanx and the corresponding to infer the typed word sequence for text input. metacarpal bone,as shown in Fig.1.Due to the fact that there is no metacarpal bone for the thumb,we use the 0 between the distal and 2 RELATED WORK proximal phalanx for the thumb.When a>ep,there is a possible To remove the dependency of physical keyboards,CamK [3]is keystroke,we set p=45 empirically. designed to input text into small devices by using a panel with 3.2.2 Detecting the possible keystroke.Based on the bending angle a keyboard layout.Microsoft Hololens [1]provides a projection of a finger,we can find the fingers which may make a keystroke. keyboard in front of the user for text input.When removing the To further determine the only finger making a keystroke,we first constraint of virtual keyboard layout,RF-glove [2]recognizes fin- compare the y-coordinate of each fingertip(the coordinate system ger movements using RF signals for mid-air interaction,while it of LMC is shown in Fig.2),and then select the finger f having the can be easily affected by the variations of environments.Consider- smallest y-coordinate as the finger making a keystroke,as shown ing the above limitations of layouts or environments,Leap Motion in Fig.1.This is because the finger pressing a key often achieves Controller [6]is introduced for mid-air HCI.Assam et al.[5]build the lowest position,when compared with other fingers.Then,we a Google Chrome extension to facilitate web browsing,but the will further verify that whether the selected finger really makes involved gestures are not suitable for text input.ATK [4]enables a keystroke or not,based on the movement trend of the finger freehand typing in the air,while it limits the number of keystrokes We compare three consecutive y-coordinates of the finger f,ie. and the number of characters in a word to be exactly the same. Different from the existing work,we aim to provide a mid-air typ- yi-1,yi.yi+1.If the finger's movement satisfies the 'V'-shaped feature,ie.,yi<yi-1 and yi<yi+1,we detect a possible keystroke ing scheme based on Leap Motion for text input while having no corresponding to the finger f. limitations on the number of keystrokes or characters in a word. 3.2.3 Removing false positive keystrokes.Finally,to remove the 3 SYSTEM DESIGN false duplicate keystrokes in a short duration,we introduce the time 3.1 System Overview difference between two consecutive keystrokes.The newly detected possible keystroke will be treated as a keystroke,only when the time Figure 2 shows the main components of the AirTyping system.The difference between the current keystroke and the last determined inputs are the coordinates of finger joints captured by Leap Motion keystroke is larger than At.According to [7],the duration of a controller,while the output is a sentence composed of inferred keystroke is usually about 185ms,thus we set At 185ms. words.While the user types in mid-air over the Leap Motion con- troller(LMC),we first utilize Keystroke Detection module to analyze 3.3 Word Inference the finger movements based on the coordinates of finger joints. and detect the possible keystroke happening.Then,based on Word After detecting the keystroke,we will infer the typed word based on Inference module,we utilize Bayesian method to infer the most the keystroke sequence.However,due to the interference of other possible word for a specific keystroke sequence,which is separated fingers'movements,these is a probability of detecting a wrong. by the space key.With the inferred words,AirTyping can output false positive or false negative keystroke.Besides,without a key- the sentence(ie.,word sequence)typed by the user for text input. board layout,we can not map the keystroke with a character(i.e., 'a''z')directly.That is to say,there is not a one-to-one mapping be- Keystroke Detection Word Inference tween the keystroke and the character.As shown in Fig.3,the user Coordinates types with standard fingering,thus the finger making a keystroke Calculating bending angles of fingers corresponds to multiple characters,e.g.,the 2nd finger corresponds eystrok The number of Keystrokes to the chars'w','s'and'x'.Therefore,to infer the typed word from Detecting the Ki the keystrokes,we introduce the Bayesian method,as described possible keystroke below. The Permutation of Keystrokes Considering that words are separated by the space key,we first P(KIW) introduce the space key,i.e.,keystrokes made by the 5th or 6th posilve keystrokes finger,to separate the keystrokes.Then,we can get a specific key- stroke sequence corresponding to a word.For convenience,we Figure 2:System Framework use Ki,i E N+to represent the keystroke sequence,while using Wi.jeN+to represent a word in the dictionary.Then,the proba- 3.2 Keystroke Detection bility P(WilKi)means that the keystroke sequence Ki is inferred as The keystroke detection module is designed to analyze the finger the word Wi.When P(WilKi)achieves the highest value,the word movements from the coordinates of finger joints,and detect the Wi is chosen as the inferred word.According to Bayes'theorem, possible keystroke happening. P(WilK:)can be calculated with Eq.(1). 3.2.1 Calculating bending angles of fingers.Intuitively,during a P(w)×P(Kw) keystroke,the fingertip first bends down,then stays at the lowest P(WjlKi)= (1) P(Ki) position for a short duration,and finally moves away.Therefore,we where P(Wi)is the prior probability representing the occurrence can detect the possible keystroke by measuring the bending angle frequency of word Wj.P(Ki)is the probability of detecting theUbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico Hao Zhang et al. keystrokes, and the one-to-many mapping between the finger and characters in standard fingering, we introduce the Bayesian method to infer the typed word sequence for text input. 2 RELATED WORK To remove the dependency of physical keyboards, CamK [3] is designed to input text into small devices by using a panel with a keyboard layout. Microsoft Hololens [1] provides a projection keyboard in front of the user for text input. When removing the constraint of virtual keyboard layout, RF-glove [2] recognizes fin￾ger movements using RF signals for mid-air interaction, while it can be easily affected by the variations of environments. Consider￾ing the above limitations of layouts or environments, Leap Motion Controller [6] is introduced for mid-air HCI. Assam et al. [5] build a Google Chrome extension to facilitate web browsing, but the involved gestures are not suitable for text input. ATK [4] enables freehand typing in the air, while it limits the number of keystrokes and the number of characters in a word to be exactly the same. Different from the existing work, we aim to provide a mid-air typ￾ing scheme based on Leap Motion for text input while having no limitations on the number of keystrokes or characters in a word. 3 SYSTEM DESIGN 3.1 System Overview Figure 2 shows the main components of the AirTyping system. The inputs are the coordinates of finger joints captured by Leap Motion controller, while the output is a sentence composed of inferred words. While the user types in mid-air over the Leap Motion con￾troller (LMC), we first utilize Keystroke Detection module to analyze the finger movements based on the coordinates of finger joints, and detect the possible keystroke happening. Then, based on Word Inference module, we utilize Bayesian method to infer the most possible word for a specific keystroke sequence, which is separated by the space key. With the inferred words, AirTyping can output the sentence (i.e., word sequence) typed by the user for text input. Ki Keystroke Detection Word Wj Keystroke Sequence Calculating bending angles of fingers Detecting the possible keystroke Removing false positive keystrokes X Y Z Coordinates Word Inference The Number of Keystrokes The Permutations of Keystrokes P(Ki|Wj ) Sentence Figure 2: System Framework. 3.2 Keystroke Detection The keystroke detection module is designed to analyze the finger movements from the coordinates of finger joints, and detect the possible keystroke happening. 3.2.1 Calculating bending angles of fingers. Intuitively, during a keystroke, the fingertip first bends down, then stays at the lowest position for a short duration, and finally moves away. Therefore, we can detect the possible keystroke by measuring the bending angle of a finger. For convenience, we introduce θ to describe the bending angle, which is formed by the distal phalanx and the corresponding metacarpal bone, as shown in Fig. 1. Due to the fact that there is no metacarpal bone for the thumb, we use the θ between the distal and proximal phalanx for the thumb. When θ > ϵb , there is a possible keystroke, we set ϵb = 45◦ empirically. 3.2.2 Detecting the possible keystroke. Based on the bending angle of a finger, we can find the fingers which may make a keystroke. To further determine the only finger making a keystroke, we first compare the y-coordinate of each fingertip (the coordinate system of LMC is shown in Fig. 2), and then select the finger f having the smallest y-coordinate as the finger making a keystroke, as shown in Fig. 1. This is because the finger pressing a key often achieves the lowest position, when compared with other fingers. Then, we will further verify that whether the selected finger really makes a keystroke or not, based on the movement trend of the finger. We compare three consecutive y-coordinates of the finger f , i.e., yi−1, yi , yi+1. If the finger’s movement satisfies the ‘V’-shaped feature, i.e., yi < yi−1 and yi < yi+1, we detect a possible keystroke corresponding to the finger f . 3.2.3 Removing false positive keystrokes. Finally, to remove the false duplicate keystrokes in a short duration, we introduce the time difference between two consecutive keystrokes. The newly detected possible keystroke will be treated as a keystroke, only when the time difference between the current keystroke and the last determined keystroke is larger than Δt. According to [7], the duration of a keystroke is usually about 185ms, thus we set Δt = 185ms. 3.3 Word Inference After detecting the keystroke, we will infer the typed word based on the keystroke sequence. However, due to the interference of other fingers’ movements, these is a probability of detecting a wrong, false positive or false negative keystroke. Besides, without a key￾board layout, we can not map the keystroke with a character (i.e., ‘a’-‘z’) directly. That is to say, there is not a one-to-one mapping be￾tween the keystroke and the character. As shown in Fig. 3, the user types with standard fingering, thus the finger making a keystroke corresponds to multiple characters, e.g., the 2nd finger corresponds to the chars ‘w’, ‘s’ and ‘x’. Therefore, to infer the typed word from the keystrokes, we introduce the Bayesian method, as described below. Considering that words are separated by the space key, we first introduce the space key, i.e., keystrokes made by the 5th or 6th finger, to separate the keystrokes. Then, we can get a specific key￾stroke sequence corresponding to a word. For convenience, we use Ki,i ∈ N+ to represent the keystroke sequence, while using Wj , j ∈ N+ to represent a word in the dictionary. Then, the proba￾bility P (Wj |Ki ) means that the keystroke sequence Ki is inferred as the word Wj . When P (Wj |Ki ) achieves the highest value, the word Wj is chosen as the inferred word. According to Bayes’ theorem, P (Wj |Ki ) can be calculated with Eq. (1), P (Wj |Ki ) = P (Wj ) × P (Ki |Wj ) P (Ki ) (1) where P (Wj ) is the prior probability representing the occurrence frequency of word Wj , P (Ki ) is the probability of detecting the