00.0) 0(0.0) represent the background.However,around the hands,there exist some interference regions,which may change the con- B4g(g,y4g） tours of fingers,resulting in detecting wrong fingertips.Thus. P+g(+g+g CamK introduces the erosion and dilation operations [17]. P(.y) Ly P-(x-4-) P(x:y) We first use the erosion operation to isolate the hands from keys and separate each finger.Then,we use the dilation (a)Fingertips (excluding thumbs) (b)A thumb operation to smooth the edge of the fingers.Fig.5(c)shows Fig.6.Features of a fingertip the optimized result of hand segmentation.Intuitively,if the In fingertip detection,we only need to detect the points color of the user's clothes is close to his/her skin color,the located in the bottom edge (from the left most point to the hand segmentation result will become worse.At this time,we right most point)of the hand,such as the blue contour of only focus on the hand region located in the keyboard area. right hand in Fig.5(d).The shape feature 0;and the positions Due to the color difference between the keyboard and human in vertical coordinates yi along the bottom edge are shown skin,CamK can still extract the hands efficiently. Fig.5(e).If we can detect five fingertips in a hand with; and yi,we will not detect the thumb specially. Otherwise,we detect the fingertip of the thumb in the right most area of left hand or left most area of right hand according to 0i and i-,i.i+.The detected fingertips of Fig.5(a) are marked in Fig.5(f). D.Keystroke Detection and Localization (a)An input image (b)Hand segmentation (c)Optimization When CamK detects the fingertips,it will track the fin- gertips to detect a possible keystroke and localize it.The keystroke localization result can be used to remove false pos- itive keystrokes.We illustrate the whole process of keystroke detection and localization together. 1)Candidate fingertip in each hand:CamK allows the (d)Fingers'contour (e)Fingertip discovery (f)Fingertips user to use all the fingers for text-entry,thus the keystroke Fig.5.Fingertip detection may be caused by the left or right hand.According to the 2)Fingertip discovery:After we extract the fingers,we observations (see section II),the fingertip (i.e.,StrokeTip) need to detect the fingertips.As shown in Fig.6(a).the pressing the key usually has the largest vertical coordinate in fingertip is usually a convex vertex of the finger.For a point that hand.Therefore,we first select the candidate fingertip with Pi(xi,yi)located in the contour of a hand,by tracing the the largest vertical coordinate in each hand.We respectively contour,we can select the point Pi(i)before P use C and C.to represent the points located in the contour and the point P+(after Pi.Here,i,q.E N.We of left hand and right hand.For a point P(,y)EC.if calculate the angle between the two vectors PP-PP+ P satisfies≥(P,(rj,y)∈C,j≠I),then乃will be according to Eq.(1).In order to simplify the calculation for, selected as the candidate fingertip in the left hand.Similarly, we map 0i in the range0a∈[0°，l80].If0:∈[a,0h],ai<0h, we can get the candidate fingertip P(r,yr)in the right hand. we call P:a candidate vertex.Considering the relative lo- In this step,we only need to get P and P to know the moving cations of the points,Pi should also satisfy yi>y and states of hands.It is unnecessary to detect other fingertips. yi>+Otherwise,Pi will not be a candidate vertex.If 2)Moving or staying:As described in the observations, there are multiple candidate vertexes,such as P in Fig.6(a), when the user presses a key,the fingertip will stay at that key for a certain duration.Therefore,we can use the loca- we choose the vertex which has the largest vertical coordinate, as P shown in Fig.6(a).Because this point has the largest tion variation of the candidate fingertip to detect a possible probability to be a fingertip.Based on extensive experiments, keystroke.In Frame i,we use P(L,y)and Pr(r,y) to represent the candidate fingertips in the left hand and right we set 0=60°，0h=150°，q=20 in this paper. hand,respectively.Based on Fig.5,the interference regions around a fingertip may affect the contour of the fingertip,there 0;arccos- PP-·PP+g (1) PP-gl·lPP+gl may exist a position deviation between the real fingertip and the detected fingertip.Therefore,if the candidate fingertips in Considering the specificity of thumbs,which may press the frame i-1,i satisfy Eq.(2),the fingertips can be treated as key (e.g.,space key)in a different way from other fingers, static,i.e.,a keystroke probably happens.Based on extensive the relative positions of P-P,P+may change.Fig.6(b) experiments,we set Ar =5 empirically. shows the thumb in the left hand.Obviously,Pi-Pi.P+ do not satisfy yi>yi-g and yi>yi+.Therefore,we use V:-4户+(-卫≤△r, (2) (i-i-)(i+)>0 to describe the relative locations of V(Em-xm-12+(r-r4-1P≤△r Pi-q,P,P+a in thumbs.Then,we choose the vertex which 3)Discovering the pressed key:For a keystroke,the finger- has the largest vertical coordinate as the fingertip. tip is located at the key and a part of the key will be visuallyrepresent the background. However, around the hands, there exist some interference regions, which may change the con￾tours of fingers, resulting in detecting wrong fingertips. Thus, CamK introduces the erosion and dilation operations [17]. We first use the erosion operation to isolate the hands from keys and separate each finger. Then, we use the dilation operation to smooth the edge of the fingers. Fig. 5(c) shows the optimized result of hand segmentation. Intuitively, if the color of the user’s clothes is close to his/her skin color, the hand segmentation result will become worse. At this time, we only focus on the hand region located in the keyboard area. Due to the color difference between the keyboard and human skin, CamK can still extract the hands efficiently. (a) An input image (b) Hand segmentation (c) Optimization (d) Fingers’contour                  3RLQWVHTXHQFH $QJOHq                  9HUWLFDOFRRUGLQDWH (e) Fingertip discovery (f) Fingertips Fig. 5. Fingertip detection 2) Fingertip discovery: After we extract the fingers, we need to detect the fingertips. As shown in Fig. 6(a), the fingertip is usually a convex vertex of the finger. For a point Pi(xi , yi) located in the contour of a hand, by tracing the contour, we can select the point Pi−q(xi−q, yi−q) before Pi and the point Pi+q(xi+q, yi+q) after Pi . Here, i, q ∈ N. We calculate the angle θi between the two vectors −−−−→ PiPi−q, −−−−→ PiPi+q, according to Eq. (1). In order to simplify the calculation for θi , we map θi in the range θi ∈ [0◦ , 180◦ ]. If θi ∈ [θl , θh], θl < θh, we call Pi a candidate vertex. Considering the relative lo￾cations of the points, Pi should also satisfy yi > yi−q and yi > yi+q. Otherwise, Pi will not be a candidate vertex. If there are multiple candidate vertexes, such as P ′ i in Fig. 6(a), we choose the vertex which has the largest vertical coordinate, as Pi shown in Fig. 6(a). Because this point has the largest probability to be a fingertip. Based on extensive experiments, we set θl = 60◦ , θh = 150◦ , q = 20 in this paper. θi = arccos −−−−→ PiPi−q · −−−−→ PiPi+q | −−−−→ PiPi−q| · |−−−−→ PiPi+q| (1) Considering the specificity of thumbs, which may press the key (e.g., space key) in a different way from other fingers, the relative positions of Pi−q, Pi , Pi+q may change. Fig. 6(b) shows the thumb in the left hand. Obviously, Pi−q, Pi , Pi+q do not satisfy yi > yi−q and yi > yi+q. Therefore, we use (xi−xi−q)·(xi−xi+q) > 0 to describe the relative locations of Pi−q, Pi , Pi+q in thumbs. Then, we choose the vertex which has the largest vertical coordinate as the fingertip. Ti ( , ) P x y i i i ( , ) P x y i q i q i q    ( , ) P x y i q i q i q    ' Pi O (0,0) x y (a) Fingertips (excluding thumbs) Ti ( , ) P x y i i i ( , ) P x y i q i q i q    ( , ) P x y i q i q i q    ' Pi O (0,0) x y (b) A thumb Fig. 6. Features of a fingertip In fingertip detection, we only need to detect the points located in the bottom edge (from the left most point to the right most point) of the hand, such as the blue contour of right hand in Fig. 5(d). The shape feature θi and the positions in vertical coordinates yi along the bottom edge are shown Fig. 5(e). If we can detect five fingertips in a hand with θi and yi−q, yi , yi+q, we will not detect the thumb specially. Otherwise, we detect the fingertip of the thumb in the right most area of left hand or left most area of right hand according to θi and xi−q, xi , xi+q. The detected fingertips of Fig. 5(a) are marked in Fig. 5(f). D. Keystroke Detection and Localization When CamK detects the fingertips, it will track the fin￾gertips to detect a possible keystroke and localize it. The keystroke localization result can be used to remove false pos￾itive keystrokes. We illustrate the whole process of keystroke detection and localization together. 1) Candidate fingertip in each hand: CamK allows the user to use all the fingers for text-entry, thus the keystroke may be caused by the left or right hand. According to the observations (see section II), the fingertip (i.e., StrokeTip) pressing the key usually has the largest vertical coordinate in that hand. Therefore, we first select the candidate fingertip with the largest vertical coordinate in each hand. We respectively use Cl and Cr to represent the points located in the contour of left hand and right hand. For a point Pl(xl , yl) ∈ Cl , if Pl satisfies yl ≥ yj (∀Pj (xj , yj ) ∈ Cl , j 6= l), then Pl will be selected as the candidate fingertip in the left hand. Similarly, we can get the candidate fingertip Pr(xr, yr) in the right hand. In this step, we only need to get Pl and Pr to know the moving states of hands. It is unnecessary to detect other fingertips. 2) Moving or staying: As described in the observations, when the user presses a key, the fingertip will stay at that key for a certain duration. Therefore, we can use the loca￾tion variation of the candidate fingertip to detect a possible keystroke. In Frame i, we use Pli (xli , yli ) and Pri (xri , yri ) to represent the candidate fingertips in the left hand and right hand, respectively. Based on Fig. 5, the interference regions around a fingertip may affect the contour of the fingertip, there may exist a position deviation between the real fingertip and the detected fingertip. Therefore, if the candidate fingertips in frame i − 1, i satisfy Eq. (2), the fingertips can be treated as static, i.e., a keystroke probably happens. Based on extensive experiments, we set ∆r = 5 empirically. p (xli − xli−1 ) 2 + (yli − yli−1 ) p 2 ≤ ∆r, (xri − xri−1 ) 2 + (yri − yri−1 ) 2 ≤ ∆r. (2) 3) Discovering the pressed key: For a keystroke, the finger￾tip is located at the key and a part of the key will be visually