distance between the camera and hands may affect the value of Frame rate selection:CamK sets the initial/default frame d(d).Therefore,for the unselected candidate fingertip (e.g.,rate of the camera to be fo=30fps (frames per second). fingertip 8 in Fig.8(d)),we do not discard it.We display its which is usually the maximal frame rate of many smartphones associated key as the candidate key.The user can select the For the ith keystroke.the number of frames containing the candidate key for text input(see Fig.1). keystroke is represented as no,.When the user has pressed Algorithm 1:Keystroke localization u keys,we can get the average number of frames during Input:Candidate fingertip set Ctip in frame i. a keystroke as o=吉：∑Tno,-In fact.refects the duration of a keystroke.When the frame rate f changes,the Remove fingertips out of the keyboard from Ctip. number of frames in a keystroke nf changes.Intuitively,a for P∈Ctip do Obtain candidate key set Ckey with four nearest keys smaller value of nif can reduce the image processing time, around P;. while a larger value of nif can improve the accuracy of keystroke localization.Based on extensive experiments (see forK,∈Ckey do if P is located in Ki then sectionV-C,we set rif=3,thusf=o·需 Calculate the coverage ratio Pk,of Kj. B.Online Calibration if pk,pr then Remove Kj from Ckey- Removing false positive keystrokes:Sometimes,the fin- else Remove Ki from Ckey: gers may keep still,even the user does not type any key. CamK may treat the non-keystroke as a keystroke by chance, if Ckey≠0then. leading to an error.Thus we introduce a temporary character Select Ki with largest pk,from Ckey. to mitigate this problem. Pi and Ki form a combination Pi,Ki>. In the process of pressing a key,the StrokeTip moves else Remove Pi from Ctip towards the key,stays at that key,and then moves away.The if Cip=0 then No keystroke occurs,return vertical coordinate of the StrokeTip first increases,then pauses, if Ctipl =1 then then decreases.If CamK has detected a keystroke in the rf Return the associated key of the only fingertip. consecutive frames,it will display the current character on For each hand,select Pi,Kj>with largest ratio Pk, the screen as a temporary character.In the next frame(s),if Use <P,K>(<P,K >to represent the fingertip the position of the StrokeTip does not satisfy the features of and its associated key in left(right)hand. a keystroke,CamK will cancel the temporary character.This Calculate d(dr)between P(P)and remaining does not have much impact on the user's experience,because fingertips in left(right)hand. of the very short time during two consecutive frames.Besides, if di>dr then Return Ki CamK also displays the candidate keys around the StrokeTip, else Return K,; the user can choose them for text input. Output:The pressed key. Movement of smartphone or keyboard:CamK presumes that the smartphone and the keyboard are kept at stable IV.OPTIMIZATIONS FOR KEYSTROKE LOCALIZATION AND positions during its usage life-cycle.For best results,we IMAGE PROCESSING recommend the user tape the paper keyboard on the panel. However,to alleviate the effect caused by the movements of A.Initial Training the mobile device or the keyboard,we offer a simple solution. Optimal parameters for image processing:For key seg- If the user uses the Delete key on the screen multiple times mentation (see section III-B2).y is used for tolerating the (e.g.,larger than 3 times),it may indicate CamK can not output change of Y caused by environments.Initially,y=50.CamK the character correctly.The movements of the device/keyboard updates+1,when the number of extracted keys may happen.Then,CamK informs the user to move his/her decreases,it stops.Then,CamK sets y to 50 and updates hands away from the keyboard for relocation.After that,the =1-1,when the number of extracted keys decreases, user can continue the typing process. it stops.In the process,when CamK gets maximum number of keys,the corresponding value is selected as the optimal C.Real Time Image Processing value for Ey. Because image processing is rather time-consuming,it is In hand segmentation,CamK uses erosion and dilation difficult to make CamK work on the mobile device.Take operations,which respectively use a kernel B [17]to process the Samsung GT-19100 smartphone as an example,when the images.In order to get the suitable size of B,the user first image size is 640*480 pixels,it needs 630ms to process this puts his/her hands on the home row of the keyboard,as shown image to localize the keystroke.When considering the time in Fig.5(a).For simplicity,we set the kernel sizes for erosion cost for taking images,processing consecutive images to track and dilation to be equal.The initial kernel size is zo =0.fingertips for keystroke detection,the time cost for localizing Then,CamK updates zi=2-1+1.When CamK can localize a keystroke will increase to 1320ms,which will lead to a very each fingertip in the correct key with zi,then CamK sets the low input speed and a bad user experience.Therefore,we kernel size as z=2i. introduce the following optimizations for CamK.distance between the camera and hands may affect the value of dl (dr). Therefore, for the unselected candidate fingertip (e.g., fingertip 8 in Fig. 8(d)), we do not discard it. We display its associated key as the candidate key. The user can select the candidate key for text input (see Fig. 1). Algorithm 1: Keystroke localization Input: Candidate fingertip set Ctip in frame i. Remove fingertips out of the keyboard from Ctip . for Pi ∈ Ctip do Obtain candidate key set Ckey with four nearest keys around Pi . for Kj ∈ Ckey do if Pi is located in Kj then Calculate the coverage ratio ρkj of Kj . if ρkj < ρl then Remove Kj from Ckey. else Remove Kj from Ckey; if Ckey 6= ∅ then Select Kj with largest ρkj from Ckey. Pi and Kj form a combination < Pi , Kj >. else Remove Pi from Ctip ; if Ctip = ∅ then No keystroke occurs, return ; if |Ctip| = 1 then Return the associated key of the only fingertip. For each hand, select < Pi , Kj > with largest ratio ρkj . Use < Pl , Kl > (< Pr, Kr >) to represent the fingertip and its associated key in left (right) hand. Calculate dl (dr) between Pl (Pr) and remaining fingertips in left (right) hand. if dl > dr then Return Kl ; else Return Kr; Output: The pressed key. IV. OPTIMIZATIONS FOR KEYSTROKE LOCALIZATION AND IMAGE PROCESSING A. Initial Training Optimal parameters for image processing: For key seg￾mentation (see section III-B2), εy is used for tolerating the change of Y caused by environments. Initially, εy = 50. CamK updates εyi = εyi−1 + 1, when the number of extracted keys decreases, it stops. Then, CamK sets εy to 50 and updates εyi = εyi−1 −1, when the number of extracted keys decreases, it stops. In the process, when CamK gets maximum number of keys, the corresponding value εyi is selected as the optimal value for εy. In hand segmentation, CamK uses erosion and dilation operations, which respectively use a kernel B [17] to process images. In order to get the suitable size of B, the user first puts his/her hands on the home row of the keyboard, as shown in Fig. 5(a). For simplicity, we set the kernel sizes for erosion and dilation to be equal. The initial kernel size is z0 = 0. Then, CamK updates zi = zi−1 + 1. When CamK can localize each fingertip in the correct key with zi , then CamK sets the kernel size as z = zi . Frame rate selection: CamK sets the initial/default frame rate of the camera to be f0 = 30fps (frames per second), which is usually the maximal frame rate of many smartphones. For the ith keystroke, the number of frames containing the keystroke is represented as n0i . When the user has pressed u keys, we can get the average number of frames during a keystroke as n¯0 = 1 u · Pi=u i=1 n0i . In fact, n¯0 reflects the duration of a keystroke. When the frame rate f changes, the number of frames in a keystroke n¯f changes. Intuitively, a smaller value of n¯f can reduce the image processing time, while a larger value of n¯f can improve the accuracy of keystroke localization. Based on extensive experiments (see section V-C), we set n¯f = 3, thus f = l f0 · n¯f n¯0 m . B. Online Calibration Removing false positive keystrokes: Sometimes, the fin￾gers may keep still, even the user does not type any key. CamK may treat the non-keystroke as a keystroke by chance, leading to an error. Thus we introduce a temporary character to mitigate this problem. In the process of pressing a key, the StrokeTip moves towards the key, stays at that key, and then moves away. The vertical coordinate of the StrokeTip first increases, then pauses, then decreases. If CamK has detected a keystroke in the n¯f consecutive frames, it will display the current character on the screen as a temporary character. In the next frame(s), if the position of the StrokeTip does not satisfy the features of a keystroke, CamK will cancel the temporary character. This does not have much impact on the user’s experience, because of the very short time during two consecutive frames. Besides, CamK also displays the candidate keys around the StrokeTip, the user can choose them for text input. Movement of smartphone or keyboard: CamK presumes that the smartphone and the keyboard are kept at stable positions during its usage life-cycle. For best results, we recommend the user tape the paper keyboard on the panel. However, to alleviate the effect caused by the movements of the mobile device or the keyboard, we offer a simple solution. If the user uses the Delete key on the screen multiple times (e.g., larger than 3 times), it may indicate CamK can not output the character correctly. The movements of the device/keyboard may happen. Then, CamK informs the user to move his/her hands away from the keyboard for relocation. After that, the user can continue the typing process. C. Real Time Image Processing Because image processing is rather time-consuming, it is difficult to make CamK work on the mobile device. Take the Samsung GT-I9100 smartphone as an example, when the image size is 640 ∗ 480 pixels, it needs 630ms to process this image to localize the keystroke. When considering the time cost for taking images, processing consecutive images to track fingertips for keystroke detection, the time cost for localizing a keystroke will increase to 1320ms, which will lead to a very low input speed and a bad user experience. Therefore, we introduce the following optimizations for CamK