Adaptively changing image sizes:We use small images C.Frame rate (e.g.,120 90 pixels)during two keystrokes to track the As described in section IV-A,the frame rate affects the fingertips,and use a large image (e.g..480 360 pixels) number of images nf during a keystroke.Obviously,with the for keystroke localization.Optimizing the large-size image larger value of nif,CamK can easily detect the keystroke and processing:When we detect a possible keystroke in(e,ye)of localize it.On the contrary,CamK may miss the keystrokes. frame i-1,then we focus on a small area Se=[P(zi,yi)Based on Fig.11,when rif >3,CamK has good performance. Sel Ti-xe≤△x,li-yel≤△y}of frame i to localize When nf >3,there is no obvious performance improvement. the keystroke.We set Ax =40,Ay=20 by default.Multi- However,increasing nf means introducing more images for thread Processing:CamK adopts three threads to detect and processing.It may increase the time latency.While considering localize the keystroke in parallel,i.e.,capturing thread to take the accuracy,false positive,and time latency,we set n =3. images,tracking thread for keystroke detection,and locating Besides,we invite 5 users to test the duration At of a thread for keystroke localization.Processing without writing keystroke.At represents the time when the StrokeTip is located and reading images:CamK directly stores the bytes of in the StrokeKey from the view of the camera.Based on the source data to the text file in binary mode,instead of Fig.12,At is usually larger than 150ms.When ri=3,the writing/reading images. frame rate is less than the maximum frame rate(30fps).CamK can work under the frame rate limitation of the smartphone V.PERFORMANCE EVALUATION Therefore,nf =3 is a suitable choice. We implement CamK on the Samsung GT-19100 smart- D.Impact of image size phone running Google's Android operating system (version We first measure the performance of CamK by adopting a 4.4.4).Samsung GT-19100 has a 2 million pixels front-facing same size for each image.Based on Fig.13,as the size of camera.We use the layout of AWK(Apple Wireless Keyboard) image increases,the performance of CamK becomes better. as the default keyboard layout,which is printed on a piece of When the size is smaller than 480 *360 pixels,CamK can US Letter sized paper.Unless otherwise specified,the frame not extract the keys correctly,the performance is rather bad. rate is 15fps,the image size is 480*460 pixels.CamK works in When the size of image is 480*360 pixels,the performance the office.We first evaluate each component of CamK.Then, is good.Keeping increasing the size does not cause obvious we invite 9 users to use CamK and compare the performance improvement.However,increasing the image size will increase of CamK with other text-entry methods the image processing time and power consumption measured A.Localization accuracy for known keystrokes by a Monsoon power monitor [18])for processing an image, as shown in Fig.14.Based on section IV-C,CamK adaptively In order to verify whether CamK has obtained the optimal change the sizes of the images.In order to guarantee high parameters for image processing,we measure the accuracy accuracy and low false positive rate,and reduce the time of keystroke localization,when CamK knows a keystroke is happening.The user presses the 59 keys(excluding the PC latency and power consumption,the size of the large image is set to 480 *380 pixels. function keys:first row,five keys in last row)on the paper In Fig.15,the size of small image decreases from 480*360 keyboard sequentially.We press each key fifty times.The to 120 90 pixels,CamK has high accuracy and low false localization result is shown in Fig.9.the localization accuracy positive rate.When the size of small images continuously is close to 100%.It means that CamK can adaptively select decreases,the accuracy decreases a lot,and the false pos- suitable values of the parameters used in image processing. itive rate increases a lot.As the image size decreases,the B.Accuracy of keystroke localization and false positive rate time cost/power consumption for locating a keystroke keeps of keystroke detection decreasing,as shown in Fig.16.Combining Fig.15 and Fig. In order to verify whether CamK can utilize the features 16,the size of small image is set to 120 *90 pixels. of a keystroke and online calibration for keystroke detection E.Time latency and power consumption and localization.We conduct the experiments in three typical Based on Fig.16,the time cost for locating a keystroke scenarios;an office environment,a coffee shop,and outdoors. is about 200ms,which is comparable to the duration of a Usually,in the office,the color of the light is close to white.In keystroke,as shown in Fig.12.It means when the user the coffee shop,the red part of light is similar to that of human stays in the pressed key,CamK can output the text without skin.In outdoors,the sunlight is basic/pure light.In each test. noticeable time latency.The time latency is within 50ms,or a user randomly makes N=500 keystrokes.Suppose CamK even smaller,which is well below human response time [10. localizes Na keystrokes correctly and treats Nf non-keystrokes In addition,we measure the power consumption of Samsung as keystrokes wrongly.We define the accuracy as pa GT-19100 smartphone in the following states:(1)idle with and the false positive rate as p-min(,1).We show the screen on;(2)writing an email;(3)keeping the camera the results of these experiments in Fig.10,which shows that on the preview mode(frame rate is 15fps);(4)running CamK CamK can achieve high accuracy (larger than 90%)with low (frame rate is 15fps)for text-entry.The power consumption false positive rate (about 5%).In the office,the localization in each state is 516mW,1189mW,1872mW,2245mW.The accuracy can achieve 95%. power consumption of CamK is a little high.Yet as a newAdaptively changing image sizes: We use small images (e.g., 120 ∗ 90 pixels) during two keystrokes to track the fingertips, and use a large image (e.g., 480 ∗ 360 pixels) for keystroke localization. Optimizing the large-size image processing: When we detect a possible keystroke in (xc, yc) of frame i−1, then we focus on a small area Sc = {Pi(xi , yi) ∈ Sc| |xi − xc| ≤ ∆x, |yi − yc| ≤ ∆y} of frame i to localize the keystroke. We set ∆x = 40, ∆y = 20 by default. Multi￾thread Processing: CamK adopts three threads to detect and localize the keystroke in parallel, i.e., capturing thread to take images, tracking thread for keystroke detection, and locating thread for keystroke localization. Processing without writing and reading images: CamK directly stores the bytes of the source data to the text file in binary mode, instead of writing/reading images. V. PERFORMANCE EVALUATION We implement CamK on the Samsung GT-I9100 smart￾phone running Google’s Android operating system (version 4.4.4). Samsung GT-I9100 has a 2 million pixels front-facing camera. We use the layout of AWK (Apple Wireless Keyboard) as the default keyboard layout, which is printed on a piece of US Letter sized paper. Unless otherwise specified, the frame rate is 15fps, the image size is 480∗460 pixels. CamK works in the office. We first evaluate each component of CamK. Then, we invite 9 users to use CamK and compare the performance of CamK with other text-entry methods. A. Localization accuracy for known keystrokes In order to verify whether CamK has obtained the optimal parameters for image processing, we measure the accuracy of keystroke localization, when CamK knows a keystroke is happening. The user presses the 59 keys (excluding the PC function keys: first row, five keys in last row) on the paper keyboard sequentially. We press each key fifty times. The localization result is shown in Fig. 9. the localization accuracy is close to 100%. It means that CamK can adaptively select suitable values of the parameters used in image processing. B. Accuracy of keystroke localization and false positive rate of keystroke detection In order to verify whether CamK can utilize the features of a keystroke and online calibration for keystroke detection and localization. We conduct the experiments in three typical scenarios; an office environment, a coffee shop, and outdoors. Usually, in the office, the color of the light is close to white. In the coffee shop, the red part of light is similar to that of human skin. In outdoors, the sunlight is basic/pure light. In each test, a user randomly makes Nk = 500 keystrokes. Suppose CamK localizes Na keystrokes correctly and treats Nf non-keystrokes as keystrokes wrongly. We define the accuracy as pa = Na Nk , and the false positive rate as pf = min( Nf Nk , 1). We show the results of these experiments in Fig. 10, which shows that CamK can achieve high accuracy (larger than 90%) with low false positive rate (about 5%). In the office, the localization accuracy can achieve 95%. C. Frame rate As described in section IV-A, the frame rate affects the number of images n¯f during a keystroke. Obviously, with the larger value of n¯f , CamK can easily detect the keystroke and localize it. On the contrary, CamK may miss the keystrokes. Based on Fig. 11, when n¯f ≥ 3, CamK has good performance. When n¯f > 3, there is no obvious performance improvement. However, increasing n¯f means introducing more images for processing. It may increase the time latency. While considering the accuracy, false positive, and time latency, we set n¯f = 3. Besides, we invite 5 users to test the duration ∆t of a keystroke. ∆t represents the time when the StrokeTip is located in the StrokeKey from the view of the camera. Based on Fig. 12, ∆t is usually larger than 150ms. When n¯f = 3, the frame rate is less than the maximum frame rate (30fps). CamK can work under the frame rate limitation of the smartphone. Therefore, n¯f = 3 is a suitable choice. D. Impact of image size We first measure the performance of CamK by adopting a same size for each image. Based on Fig. 13, as the size of image increases, the performance of CamK becomes better. When the size is smaller than 480 ∗ 360 pixels, CamK can not extract the keys correctly, the performance is rather bad. When the size of image is 480 ∗ 360 pixels, the performance is good. Keeping increasing the size does not cause obvious improvement. However, increasing the image size will increase the image processing time and power consumption ( measured by a Monsoon power monitor [18]) for processing an image, as shown in Fig. 14. Based on section IV-C, CamK adaptively change the sizes of the images. In order to guarantee high accuracy and low false positive rate, and reduce the time latency and power consumption, the size of the large image is set to 480 ∗ 380 pixels. In Fig. 15, the size of small image decreases from 480∗360 to 120 ∗ 90 pixels, CamK has high accuracy and low false positive rate. When the size of small images continuously decreases, the accuracy decreases a lot, and the false pos￾itive rate increases a lot. As the image size decreases, the time cost/power consumption for locating a keystroke keeps decreasing, as shown in Fig. 16. Combining Fig. 15 and Fig. 16, the size of small image is set to 120 ∗ 90 pixels. E. Time latency and power consumption Based on Fig. 16, the time cost for locating a keystroke is about 200ms, which is comparable to the duration of a keystroke, as shown in Fig. 12. It means when the user stays in the pressed key, CamK can output the text without noticeable time latency. The time latency is within 50ms, or even smaller, which is well below human response time [10]. In addition, we measure the power consumption of Samsung GT-I9100 smartphone in the following states: (1) idle with the screen on; (2) writing an email; (3) keeping the camera on the preview mode (frame rate is 15fps); (4) running CamK (frame rate is 15fps) for text-entry. The power consumption in each state is 516mW, 1189mW, 1872mW, 2245mW. The power consumption of CamK is a little high. Yet as a new