This article has been accepted for publication in a future issue of this journal,but has not been fully edited.Content may change prior to final publication.Citation information:DOI 10.1109/JIOT.2021.3114224.IEEE Internet of Things Journal IEEE INTERNET OF THINGS JOURNAL,VOL.XX,NO.XX,XX 2021 11 (a)Keystroke oc vs.differ(b)Keystroke oo vs.differ (c)Keystroke localization vs.differ- (d)Keystroke localization vs.differ- ent light sources. ent surfaces. ent devices. ent keyboard layouts. Fig.16.Keystroke localization performance under different complex scenarios. In addition,to show the efficiency of our frame-skipping 100 scheme in Section IV-El,we evaluate how the frame rates affect the system performance.Specifically,the default frame 0 rate of the camera is 30 fps,which is usually the maxi- mal/default frame rate of off-the-shelf Android smartphones. Then,we change the interval of processing an image,i.e.,we 23 process every Na images and Na E[1,10].Fig.14(d)shows that the change of Na has little effect on the performance of key tracking.However,Na affects the frequency of key Fig.17.Keystroke localization vs.backgrounds with different stripes tracking and keystroke localization performance.Fig.15(d) shows that when the interval Na <=5,DynaKey performs well in keystroke localization.When the interval is 5,the lo- stripes which are similar to lines in keyboard.Therefore,we calization accuracy reaches 95.5%.However,when the interval use four kinds of backgrounds with stripes(i.e.,grid,pinstripe, keeps increasing,it may miss some keystrokes,leading to a wood with light color,wood with deep color)to evaluate lower localization accuracy and a higher false negative rate.To DynaKey.As shown in Fig.17,the stripes in backgrounds achieve a better trade-off between the keystroke localization only have a little effect on keystroke detection and localization. performance and computation overhead,we set the interval In fact,the lines in keyboard have fixed rules (e.g.,distance, Na =5,i.e.,DynaKey processes every 5 frames. length),while the rules in stripes of backgrounds are usually different from that in keyboard.Thus DynaKey can work E.Effect of Complex Scenarios well for keyboard/key tracking.In regard to the performance 1)Different Light Sources:In this experiment,we evaluate decrease in the fourth bar,it is mainly caused by the color whether DynaKey can locate the keystrokes efficiently in the of background,which is closer to skin color and can affect environments with different light conditions.We conduct the hand/finger extraction. experiments in three typical scenarios:an office environment 4)Different Devices:In addition to the Samsung Galaxy (light color is close to white),outdoors (basic light).and a S9 smartphone,we also evaluate the performance of keystroke restaurant (light is a bit warm).A subject is instructed to localization in DynaKey using two other smartphones-XiaoMi type the same set of characters in these three scenarios.As Note 3(Android OS 8.1)and Huawei Honor 7i (Android OS show in Fig.16(a),DynaKey achieves good performance in all 6.0).As shown in Fig.16(c),the average localization accuracy the three scenarios,i.e.,the localization accuracy is 94.4%on is 95.5%for Samsung phone,94.7%for XiaoMi phone, average,while the average localization error,false positive rate and false negative rate are 2.1%,2.3%and 3.4%,respectively. and 92.9%for Huawei phone.The performance difference In the office scenario,DynaKey achieves the best keystroke may come from the aspects like the location of camera, localization accuracy,i.e.,95.5%. viewing angle of camera,the size of device,etc.Nevertheless, 2)Different Surfaces:In real-world applications,repeat- DynaKey can work well in different devices. edly handling a printed paper keyboard may easily cause 5)Different Keyboard Layouts:In this experiment,we use warping.In this experiment,we use a flat paper keyboard and two common keyboard layouts-Hololens [13]and US ANSI a wrinkled one to evaluate the effect of wrinkled paper on [17],to evaluate the performance of keystroke localization keystroke localization.Fig.16(b)shows that when typing on Each layout is printed on a piece of A4-sized paper.Fig.16(d) the flat keyboard layout,the accuracy of keystroke localization shows that whatever the keyboard layout is,DynaKey has achieves 95.5%on average.When typing on the wrinkled one, good performance in keystroke localization,i.e.,the accuracy the localization accuracy is 93.6%on average.Wrinkled paper achieves above 94.3%.Besides,to further explore the scal- may affect the detection of long lines and corner points of the ability of DynaKey,we draw a A4-sized Hololens keyboard keyboard layout,thus the keystroke localization performance layout on the surface of a table,and the subject is instructed to for a wrinkled paper keyboard is slightly worse than that of type a set of characters as in previous experiments.As shown the flat one. in Fig.16(d),even if we replace the paper keyboard with a 3)Different Backgrounds:When placing the keyboard on drawn keyboard layout,the accuracy of keystroke localization object surface,the texture of surface (i.e.,background)may still reaches above 93%,indicating that DynaKey can work affect the detection of keyboard,especially for surfaces with with a simple keyboard layout.2327-4662 (c) 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JIOT.2021.3114224, IEEE Internet of Things Journal IEEE INTERNET OF THINGS JOURNAL, VOL. XX, NO. XX, XX 2021 11 94.9 2 2.3 3.1 95.5 1.9 2.1 2.6 92.9 2.5 2.5 4.6 outdoor office restaurant Different light sources 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate (a) Keystroke localization vs. differ￾ent light sources. 95.5 1.9 2.1 2.6 93.6 2.5 2.3 3.9 flat winkled Different surfaces 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate (b) Keystroke localization vs. differ￾ent surfaces. 95.5 1.9 2.1 2.6 94.7 2.4 2.4 2.9 92.9 2.7 3.4 4.4 Samsung Xiaomi Huawei Different devices 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate (c) Keystroke localization vs. differ￾ent devices. 95.5 1.9 2.1 2.6 95 2.8 2.4 2.2 93.7 2.6 3.9 3.7 Hololens US ANSI Drawn Different keyboard layouts 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate (d) Keystroke localization vs. differ￾ent keyboard layouts. Fig. 16. Keystroke localization performance under different complex scenarios. In addition, to show the efficiency of our frame-skipping scheme in Section IV-E1, we evaluate how the frame rates affect the system performance. Specifically, the default frame rate of the camera is 30 fps, which is usually the maxi￾mal/default frame rate of off-the-shelf Android smartphones. Then, we change the interval of processing an image, i.e., we process every Nd images and Nd ∈ [1, 10]. Fig. 14(d) shows that the change of Nd has little effect on the performance of key tracking. However, Nd affects the frequency of key tracking and keystroke localization performance. Fig. 15(d) shows that when the interval Nd <= 5, DynaKey performs well in keystroke localization. When the interval is 5, the lo￾calization accuracy reaches 95.5%. However, when the interval keeps increasing, it may miss some keystrokes, leading to a lower localization accuracy and a higher false negative rate. To achieve a better trade-off between the keystroke localization performance and computation overhead, we set the interval Nd = 5, i.e., DynaKey processes every 5 frames. E. Effect of Complex Scenarios 1) Different Light Sources: In this experiment, we evaluate whether DynaKey can locate the keystrokes efficiently in the environments with different light conditions. We conduct the experiments in three typical scenarios: an office environment (light color is close to white), outdoors (basic light), and a restaurant (light is a bit warm). A subject is instructed to type the same set of characters in these three scenarios. As show in Fig. 16(a), DynaKey achieves good performance in all the three scenarios, i.e., the localization accuracy is 94.4% on average, while the average localization error, false positive rate and false negative rate are 2.1%, 2.3% and 3.4%, respectively. In the office scenario, DynaKey achieves the best keystroke localization accuracy, i.e., 95.5%. 2) Different Surfaces: In real-world applications, repeat￾edly handling a printed paper keyboard may easily cause warping. In this experiment, we use a flat paper keyboard and a wrinkled one to evaluate the effect of wrinkled paper on keystroke localization. Fig. 16(b) shows that when typing on the flat keyboard layout, the accuracy of keystroke localization achieves 95.5% on average. When typing on the wrinkled one, the localization accuracy is 93.6% on average. Wrinkled paper may affect the detection of long lines and corner points of the keyboard layout, thus the keystroke localization performance for a wrinkled paper keyboard is slightly worse than that of the flat one. 3) Different Backgrounds: When placing the keyboard on object surface, the texture of surface (i.e., background) may affect the detection of keyboard, especially for surfaces with 92.6 4.2 3.7 3.2 93.9 3.4 2.7 2.7 92.9 4.5 3.9 2.6 91.8 3.3 3.8 4.9 Grid Pinstripe Light-wood Deep-wood Backgrounds 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate Fig. 17. Keystroke localization vs. backgrounds with different stripes stripes which are similar to lines in keyboard. Therefore, we use four kinds of backgrounds with stripes (i.e., grid, pinstripe, wood with light color, wood with deep color) to evaluate DynaKey. As shown in Fig. 17, the stripes in backgrounds only have a little effect on keystroke detection and localization. In fact, the lines in keyboard have fixed rules (e.g., distance, length), while the rules in stripes of backgrounds are usually different from that in keyboard. Thus DynaKey can work well for keyboard/key tracking. In regard to the performance decrease in the fourth bar, it is mainly caused by the color of background, which is closer to skin color and can affect hand/finger extraction. 4) Different Devices: In addition to the Samsung Galaxy S9 smartphone, we also evaluate the performance of keystroke localization in DynaKey using two other smartphones–XiaoMi Note 3 (Android OS 8.1) and Huawei Honor 7i (Android OS 6.0). As shown in Fig. 16(c), the average localization accuracy is 95.5% for Samsung phone, 94.7% for XiaoMi phone, and 92.9% for Huawei phone. The performance difference may come from the aspects like the location of camera, viewing angle of camera, the size of device, etc. Nevertheless, DynaKey can work well in different devices. 5) Different Keyboard Layouts: In this experiment, we use two common keyboard layouts–Hololens [13] and US ANSI [17], to evaluate the performance of keystroke localization. Each layout is printed on a piece of A4-sized paper. Fig. 16(d) shows that whatever the keyboard layout is, DynaKey has good performance in keystroke localization, i.e., the accuracy achieves above 94.3%. Besides, to further explore the scal￾ability of DynaKey, we draw a A4-sized Hololens keyboard layout on the surface of a table, and the subject is instructed to type a set of characters as in previous experiments. As shown in Fig. 16(d), even if we replace the paper keyboard with a drawn keyboard layout, the accuracy of keystroke localization still reaches above 93%, indicating that DynaKey can work with a simple keyboard layout. Authorized licensed use limited to: Nanjing University. Downloaded on December 03,2021 at 08:56:41 UTC from IEEE Xplore. Restrictions apply