This article has been accepted for publication in a future issue of this journal,but has not been fully edited.Content may change prior to final publication.Citation information:DOI 10.1109/JIOT.2021.3114224.IEEE Internet of Things Journal IEEE INTERNET OF THINGS JOURNAL,VOL.XX,NO.XX,XX 2021 12 , 100 80 Localization accuracy 60 Localization error 40,·· 60 False poitve rate False negalve rale 20 150 0 50 100 0 100 2 4 5 0 6 7 8 9 10 1112 Image number Keystroke numbe Different subjects (a)The latency of key tracking. (b)The latency of keystroke localiza- (a) ocalization pe vs.provided text input for different subjects tion. Fig.18.The latency of key tracking and keystroke localization. F.Latency and Power Consumption We first evaluate the time delay of key tracking for a frame Specifically,we compute the time duration from getting an input image to obtaining the updated coordinates of keys 6 8 9 10 11 12 in 100 images.Fig.18(a)shows a large variation in the Different subjects distribution of latency.This may be due to the difference in (b) Localization performance vs.self-determined input for different subjects the number of pixels involved in keypoint selection.However, Our the average time cost for tracking keys for one frame is 26 ms,which is smaller than the inter-frame duration,i.e..33ms. It indicates that our key tracking scheme satisfies the real- time requirement for real-world applications.To evaluate the latency of processing a keystroke,we conduct the experiment by randomly pressing 100 keystrokes,and calculate the time 1 2 6 10 11 Different subjects duration from detecting a possible keystroke to locating the (c)Input speed vs.keyboard for different subjects pressed key.Fig.18(b)shows that the mean latency is 63 ms, which is below human response time [23].Overall,with low Fig.19.Performance of different subjects. latency for key tracking and keystroke localization,DynaKey an experiment,a subject types a set of predefined text and the provides a real-time text input method for head-mounted self-determined text naturally.Each experiment lasts for two devices. hours with 3 breaks in due course. To measure the power consumption of DynaKey on the 1)Text Input:Each subject is first instructed to type a Samsung Galaxy S9 smartphone,we use Battery Historian predefined words (1600 characters).Fig.19(a)show that [12]by Google.For comparison,we measure the average DynaKey performs well,i.e.,achieving 95.0%localization power consumption in four different states:1)idle,with accuracy on average,while the localization accuracy of 'sub- the screen off;2)idle,with the screen on;3)keeping the ject 7'achieves 96%.The average localization error,false camera on the preview mode;4)running our system for positive rate and false negative rate are 2.3%,2.9%and 2.7%, text input.As shown in Table I,by comparing the case of respectively.They are then allowed to type a self-determined 'Backlight','Camera-on'and 'DynaKey',more than 64%of words (2000 characters).Fig.19(b)shows that the average power consumption comes from camera,which is essential for localization accuracy is 94.7%,and the average localization camera-based applications.The additional power consumption error,false positive rate and false negative rate are 2.4%,3.3% by DynaKey is 1092mW,i.e.,DynaKey consumes about 25% and 2.9%,respectively.The performance is comparable to that additional power.In future,we will optimize the approach to of typing predefined text. further reduce the power consumption. 2)Input Speed:We also evaluate the input speeds of differ- TABLE I ent subjects in terms of WPM(words per minute).Subjects are POWER CONSUMPTION instructed to type the predefined characters with the proposed DynaKey system and Microsoft Hololens 1 [13],respectively. Power Idle 39.2+14.7 mW (CPU only) Fig.19(c)shows that the average text input speed of DynaKey Backlight 464.7+6.7 mW (CPU Screen) is 13.8 WPM while that of Hololens 1 is 6.4 WPM.Although Camera-on 3186.8+12.6 mW (CPU+Screen+Camera) the result of DynaKey is slower than that of typing on a DynaKey 4278.6±25.9mW(Total) physical keyboard [9].it still achieves 2X typing speedup compared to Hololens 1.In addition,typing with Hololens G.Evaluation on Text Input usually requires head movements for key selection,which may To further evaluate the performance of DynaKey on text lead to the fatigue of head. input,we invite twelve volunteers and instruct them to type 3)Difference with the State-of-the-art Text Input Methods: on the virtual keyboard.The typed word sequences are picked In stationary scenarios,UbiK [28]and Camk [29]are typical from the top-5000 words of the word frequency corpus [32], state-of-the-art text input methods for mobile devices.In terms which well represents the usage frequency of English words. of text input performance,these two approaches and our Before evaluation,all subjects have 30-minute trial runs.In DynaKey have comparable keystroke localization accuracy2327-4662 (c) 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JIOT.2021.3114224, IEEE Internet of Things Journal IEEE INTERNET OF THINGS JOURNAL, VOL. XX, NO. XX, XX 2021 12 1 2 3 4 5 6 7 8 9 10 The duration Nd 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate interval 0 50 100 The number of keystrokes 40 50 60 70 80 Keystroke delay (ms) Keystroke number 0 50 100 The number of jitters 0 20 40 60 80 100 Tracking delay (ms) Image number (a) The latency of key tracking. 1 2 3 4 5 6 7 8 9 10 The duration Nd 0 20 40 60 80 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate interval 0 50 100 The number of keystrokes 40 50 60 70 80 Keystroke delay (ms) Keystroke number 0 50 100 The number of jitters 0 20 40 60 80 100 Tracking delay (ms) Image number (b) The latency of keystroke localiza￾tion. Fig. 18. The latency of key tracking and keystroke localization. F. Latency and Power Consumption We first evaluate the time delay of key tracking for a frame. Specifically, we compute the time duration from getting an input image to obtaining the updated coordinates of keys in 100 images. Fig. 18(a) shows a large variation in the distribution of latency. This may be due to the difference in the number of pixels involved in keypoint selection. However, the average time cost for tracking keys for one frame is 26 ms, which is smaller than the inter-frame duration, i.e., 33ms. It indicates that our key tracking scheme satisfies the real￾time requirement for real-world applications. To evaluate the latency of processing a keystroke, we conduct the experiment by randomly pressing 100 keystrokes, and calculate the time duration from detecting a possible keystroke to locating the pressed key. Fig. 18(b) shows that the mean latency is 63 ms, which is below human response time [23]. Overall, with low latency for key tracking and keystroke localization, DynaKey provides a real-time text input method for head-mounted devices. To measure the power consumption of DynaKey on the Samsung Galaxy S9 smartphone, we use Battery Historian [12] by Google . For comparison, we measure the average power consumption in four different states: 1) idle, with the screen off; 2) idle, with the screen on; 3) keeping the camera on the preview mode; 4) running our system for text input. As shown in Table I, by comparing the case of ‘Backlight’, ‘Camera-on’ and ‘DynaKey’, more than 64% of power consumption comes from camera, which is essential for camera-based applications. The additional power consumption by DynaKey is 1092mW, i.e., DynaKey consumes about 25% additional power. In future, we will optimize the approach to further reduce the power consumption. TABLE I POWER CONSUMPTION Power Idle 39.2 ± 14.7 mW (CPU only) Backlight 464.7 ± 6.7 mW (CPU + Screen) Camera-on 3186.8 ± 12.6 mW (CPU+Screen+Camera) DynaKey 4278.6 ± 25.9 mW (Total) G. Evaluation on Text Input To further evaluate the performance of DynaKey on text input, we invite twelve volunteers and instruct them to type on the virtual keyboard. The typed word sequences are picked from the top-5000 words of the word frequency corpus [32], which well represents the usage frequency of English words. Before evaluation, all subjects have 30-minute trial runs. In 93.8 3.82.52.4 94.1 2.44.43.5 95.7 2.63.21.7 95.5 2.62.51.9 94.6 2.32.33.1 95.7 2.32.9 2 96.1 1.62.22.3 95 1.92.53.1 95.5 2.12.82.4 94 2.53.43.5 95.5 1.82.92.7 94.1 2.43.83.5 1 2 3 4 5 6 7 8 9 10 11 12 Different subjects 0 50 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate (a) Localization performance vs. provided text input for different subjects 94.1 2.93.1 3 93.9 2.73.23.4 95.2 2 3.52.8 95.2 2.23.42.6 95 2.12.82.9 94.8 2.13.33.1 94.9 2.43.32.7 95.3 1.93.22.8 94.4 2.53.13.1 94.6 2.74.22.7 94.2 2.73.83.1 94.7 2.32.9 3 1 2 3 4 5 6 7 8 9 10 11 12 Different subjects 0 50 100 Percentage (%) Localization accuracy Localization error False positive rate False negative rate (b) Localization performance vs. self-determined input for different subjects 1 2 3 4 5 6 7 8 9 10 11 12 Different subjects 0 5 10 15 Words per minnute Our system Hololens (c) Input speed vs. keyboard for different subjects Fig. 19. Performance of different subjects. an experiment, a subject types a set of predefined text and the self-determined text naturally. Each experiment lasts for two hours with 3 breaks in due course. 1) Text Input: Each subject is first instructed to type a predefined words (1600 characters). Fig. 19(a) show that DynaKey performs well, i.e., achieving 95.0% localization accuracy on average, while the localization accuracy of ‘sub￾ject 7’ achieves 96%. The average localization error, false positive rate and false negative rate are 2.3%, 2.9% and 2.7%, respectively. They are then allowed to type a self-determined words (2000 characters). Fig. 19(b) shows that the average localization accuracy is 94.7%, and the average localization error, false positive rate and false negative rate are 2.4%, 3.3% and 2.9%, respectively. The performance is comparable to that of typing predefined text. 2) Input Speed: We also evaluate the input speeds of differ￾ent subjects in terms of WPM (words per minute). Subjects are instructed to type the predefined characters with the proposed DynaKey system and Microsoft Hololens 1 [13], respectively. Fig. 19(c) shows that the average text input speed of DynaKey is 13.8 WPM while that of Hololens 1 is 6.4 WPM. Although the result of DynaKey is slower than that of typing on a physical keyboard [9], it still achieves 2X typing speedup compared to Hololens 1. In addition, typing with Hololens usually requires head movements for key selection, which may lead to the fatigue of head. 3) Difference with the State-of-the-art Text Input Methods: In stationary scenarios, UbiK [28] and Camk [29] are typical state-of-the-art text input methods for mobile devices. In terms of text input performance, these two approaches and our DynaKey have comparable keystroke localization accuracy Authorized licensed use limited to: Nanjing University. Downloaded on December 03,2021 at 08:56:41 UTC from IEEE Xplore. Restrictions apply