YIN ETAL:CAMK:CAMERA-BASED KEYSTROKE DETECTION AND LOCALIZATION FOR SMALL MOBILE DEVICES 2237 Camera based on the location of the touch.These wearable keyboards often need the user to wear devices around the hands or fin- gers,thus leading to the decrease of user experience. On-Screen Keyboards.On-screen keyboards allow the user to enter characters on a touch screen.Considering the lim- ited area of the keyboard on the screen,BigKey [3]and ZoomBoard [4]adaptively change the size of keys.Context- Type [16]leverages hand postures to improve mobile touch Fig.1.A typical use case of CamK screen text entry.Kwon et al.[17]introduce the regional error correction method to reduce the number of necessary other resources like audio signals,CamK should detect touches.ShapeWriter [18]recognizes a word based on the keystrokes only with images.To address this challenge, trace over successive letters in the word.Sandwich key- CamK combines keystroke detection with keystroke locali- board [19]affords ten-finger touch typing by utilizing a zation.For a potential keystroke,if there is no valid key touch sensor on the back side of a device.Usually,on-screen pressed by the fingertip,CamK will remove the keystroke keyboards occupy the screen area and support only one fin- and recognize it as a non-keystroke.Additionally,CamK ger for typing.Besides,it often needs to switch between dif- introduces online calibration,i.e.,using the movement fea- ferent screens to type letters,digits,punctuations,etc. tures of the fingertip after a keystroke,to further decrease Projection Keyboards.Projection keyboards usually need a the false positive rate. visible light projector or lasers to cast a keyboard,and then (3)Processing Latency:To serve as a text-entry method, utilize image processing methods [5]or infrared light [6]to when the user presses a key on the paper keyboard,CamK detect the typing events.Hu et al.use a pico-projector to should output the character of the key without any noticeable project the keyboard on the table,and then detect the touch latency.However,due to the limited computing resources of interaction by the distortion of the keyboard projection [201. small mobile devices,the heavy computation overhead of Roeber et al.utilize a pattern projector to display the key- image processing will lead to a large latency.To address this board layout on the flat surface,and then detect the key- challenge,CamK optimizes the computation-intensive mod- board events based on the intersection of fingers and ules by adaptively changing image sizes,focusing on the tar- infrared light [21].The projection keyboard often requires get area in the large-size image,adopting multiple threads the extra equipments,e.g.,a visible light projector,infrared and removing the operations of writing/reading images. light modules,etc.The extra equipments increase the cost We make the following contributions in this paper(a pre- and introduce the inconvenience of text entry. liminary version of this work appeared in [101) Camera Based Keyboards.Camera based virtual keyboards use the captured image [22]or video [23]to infer the key- We design a practical framework for CamK,which stroke.Gesture keyboard [22]gets the input by recognizing operates using a smart mobile device camera and a the finger's gesture.It works without a keyboard layout,thus portable paper keyboard.Based on image process- the user needs to remember the mapping between the keys ing,CamK can detect and locate the keystroke with and the finger's gestures.Visual Panel [8]works with a high accuracy and low false positive rate. printed keyboard on a piece of paper.It requires the user to We realize real time text-entry for small mobile use only one finger and wait for one second before each key- devices with limited resources,by optimizing the stroke.Malik et al.present the Visual Touchpad [24]to track computation-intensive modules.Additionally,we the 3D positions of the fingertips based on two downward- introduce word prediction to further improve the pointing cameras and a stereo.Adajania et al.[9]detect the input speed and reduce the error rate. keystroke based on shadow analysis with a standard web We implement CamK on smartphones running camera.Hagara et al.estimate the finger positions and detect Android.We first evaluate each module in CamK. clicking events based on edge detection,fingertip localization Then,we conduct extensive experiments to test the performance of CamK.After that,we compare CamK etc [251.In regard to the iPhone app paper keyboard [261, which only allows the user to use one finger to input letters. with other methods in input speed and error rate. The above research work usually focuses on detecting and 2 RELATED WORK tracking the fingertips,instead of locating the fingertip in a key's area of the keyboard,which is researched in our paper. Considering the small sizes of mobile devices,a lot of virtual In addition to the above text-entry solutions,MacKenzie keyboards are proposed for text entry,e.g.,wearable key- et al.[27]describe the text entry for mobile computing. boards,on-screen keyboards,projection keyboards,camera Zhang et al.[28]propose Okuli to locate user's finger based based keyboards,etc. on visible light communication modules,LED,and light Wearable Keyboards.Wearable keyboards sense and recog- sensors.Wang et al.[7]propose UbiK to locate the keystroke nize the typing behavior based on the sensors built into rings based on audio signals.The existing work usually needs [1],[11],gloves [121,and so on.TypingRing [13]utilizes the extra equipments,or only allows one finger to type,or embedded sensors of the ring to input text.Finger-Joint key- needs to change the user's typing behavior,while difficult pad [14]works with a glove equipped with the pressure sen- to provide a PC-like text-entry experience.In this paper,we sors.The Senseboard [2]consists of two rubber pads and propose a text-entry method based on the built-in camera of senses the movements in the palm to get keystrokes.Funk the mobile device and a paper keyboard,to provide a simi- et al.[15]utilize a touch sensitive wristband to enter text lar user experience to using physical keyboards.other resources like audio signals, CamK should detect keystrokes only with images. To address this challenge, CamK combines keystroke detection with keystroke locali￾zation. For a potential keystroke, if there is no valid key pressed by the fingertip, CamK will remove the keystroke and recognize it as a non-keystroke. Additionally, CamK introduces online calibration, i.e., using the movement fea￾tures of the fingertip after a keystroke, to further decrease the false positive rate. (3) Processing Latency: To serve as a text-entry method, when the user presses a key on the paper keyboard, CamK should output the character of the key without any noticeable latency. However, due to the limited computing resources of small mobile devices, the heavy computation overhead of image processing will lead to a large latency. To address this challenge, CamK optimizes the computation-intensive mod￾ules by adaptively changing image sizes, focusing on the tar￾get area in the large-size image, adopting multiple threads and removing the operations of writing/reading images. We make the following contributions in this paper (a pre￾liminary version of this work appeared in [10]).
We design a practical framework for CamK, which operates using a smart mobile device camera and a portable paper keyboard. Based on image process￾ing, CamK can detect and locate the keystroke with high accuracy and low false positive rate.
We realize real time text-entry for small mobile devices with limited resources, by optimizing the computation-intensive modules. Additionally, we introduce word prediction to further improve the input speed and reduce the error rate.
We implement CamK on smartphones running Android. We first evaluate each module in CamK. Then, we conduct extensive experiments to test the performance of CamK. After that, we compare CamK with other methods in input speed and error rate. 2 RELATED WORK Considering the small sizes of mobile devices, a lot of virtual keyboards are proposed for text entry, e.g., wearable key￾boards, on-screen keyboards, projection keyboards, camera based keyboards, etc. Wearable Keyboards. Wearable keyboards sense and recog￾nize the typing behavior based on the sensors built into rings [1], [11], gloves [12], and so on. TypingRing [13] utilizes the embedded sensors of the ring to input text. Finger-Joint key￾pad [14] works with a glove equipped with the pressure sen￾sors. The Senseboard [2] consists of two rubber pads and senses the movements in the palm to get keystrokes. Funk et al. [15] utilize a touch sensitive wristband to enter text based on the location of the touch. These wearable keyboards often need the user to wear devices around the hands or fin￾gers, thus leading to the decrease of user experience. On-Screen Keyboards. On-screen keyboards allow the user to enter characters on a touch screen. Considering the lim￾ited area of the keyboard on the screen, BigKey [3] and ZoomBoard [4] adaptively change the size of keys. Context￾Type [16] leverages hand postures to improve mobile touch screen text entry. Kwon et al. [17] introduce the regional error correction method to reduce the number of necessary touches. ShapeWriter [18] recognizes a word based on the trace over successive letters in the word. Sandwich key￾board [19] affords ten-finger touch typing by utilizing a touch sensor on the back side of a device. Usually, on-screen keyboards occupy the screen area and support only one fin￾ger for typing. Besides, it often needs to switch between dif￾ferent screens to type letters, digits, punctuations, etc. Projection Keyboards. Projection keyboards usually need a visible light projector or lasers to cast a keyboard, and then utilize image processing methods [5] or infrared light [6] to detect the typing events. Hu et al. use a pico-projector to project the keyboard on the table, and then detect the touch interaction by the distortion of the keyboard projection [20]. Roeber et al. utilize a pattern projector to display the key￾board layout on the flat surface, and then detect the key￾board events based on the intersection of fingers and infrared light [21]. The projection keyboard often requires the extra equipments, e.g., a visible light projector, infrared light modules, etc. The extra equipments increase the cost and introduce the inconvenience of text entry. Camera Based Keyboards. Camera based virtual keyboards use the captured image [22] or video [23] to infer the key￾stroke. Gesture keyboard [22] gets the input by recognizing the finger’s gesture. It works without a keyboard layout, thus the user needs to remember the mapping between the keys and the finger’s gestures. Visual Panel [8] works with a printed keyboard on a piece of paper. It requires the user to use only one finger and wait for one second before each key￾stroke. Malik et al. present the Visual Touchpad [24] to track the 3D positions of the fingertips based on two downward￾pointing cameras and a stereo. Adajania et al. [9] detect the keystroke based on shadow analysis with a standard web camera. Hagara et al. estimate the finger positions and detect clicking events based on edge detection, fingertip localization, etc [25]. In regard to the iPhone app paper keyboard [26], which only allows the user to use one finger to input letters. The above research work usually focuses on detecting and tracking the fingertips, instead of locating the fingertip in a key’s area of the keyboard, which is researched in our paper. In addition to the above text-entry solutions, MacKenzie et al. [27] describe the text entry for mobile computing. Zhang et al. [28] propose Okuli to locate user’s finger based on visible light communication modules, LED, and light sensors. Wang et al. [7] propose UbiK to locate the keystroke based on audio signals. The existing work usually needs extra equipments, or only allows one finger to type, or needs to change the user’s typing behavior, while difficult to provide a PC-like text-entry experience. In this paper, we propose a text-entry method based on the built-in camera of the mobile device and a paper keyboard, to provide a simi￾lar user experience to using physical keyboards. Fig. 1. A typical use case of CamK. YIN ET AL.: CAMK: CAMERA-BASED KEYSTROKE DETECTION AND LOCALIZATION FOR SMALL MOBILE DEVICES 2237