44:4 Y.Yin et al. and evaluation of an approach to infer eating moments using a 3-axis accelerometer in a smart- watch.Xu et al.[35]build a classifier to identify users'hand and finger gestures utilizing the essential features of accelerometer and gyroscope data measured from a smartwatch.Huang et al. [18]build a system to monitor brushing quality using a manual toothbrush modified by attaching small magnets to the handle and an off-the-shelf smartwatch.These approaches typically extract features from sensor data and apply machine learning techniques for gesture recognition. In-air gesture tracking:Zhou et al.[42-44]utilize a kinematic chain to track human upper- limb motion by placing multiple devices on the arm.Cutti et al.[11]utilize the joint angles to track the movements of upper limbs by placing sensors on the chest,shoulder,arm,and wrist. Chen et al.[8]design a wearable system consisting of a pair of magnetometers on fingers and a permanent magnet affixed to the thumb and introduce uTrack to convert the thumb and fingers into a continuous input system (e.g.,3D pointing).Shen et al.[29]utilize the 5-DoF arm model and HMM to track the 3D posture of the arm,using both motion and magnetic sensors in a smartwatch. In fact,accurate in-air gesture tracking in real time can be very challenging.Besides,obtaining the 3D moving trajectory does not mean recognizing in-air gestures.In this article,we do not require accurate trajectory tracking while aiming to obtain gesture contour and recognize it as a character. Writing in the air:Zhang et al.[39]quantify data into small integral vectors based on accel- eration orientation and then use HMM to recognize 10 Arabic numerals.Wang et al.[32]present IMUPEN to reconstruct motion trajectory and recognize handwritten digits.Bashir et al.[6]use a pen equipped with inertial sensors and apply DTW to recognize handwritten characters.Agrawal et al.[1]recognize handwritten capital letters and Arabic numerals in a 2D plane based on strokes and a grammar tree by using the built-in accelerometer in smartphone.Amma et al.[2]design a glove equipped with inertial sensors and use SVM,HMM,and statistical language model to rec- ognize capital letters,sentences,and so on.Deselaers et al.[13]present GyroPen to reconstruct the writing path for pen-like interaction.Xu et al.[36]utilize the continuous density HMM and Viterbi algorithm to recognize handwritten digits and letters using inertial sensors.In this article, we focus on single in-air character recognition without the assistance of a language model.For a character,we do not define specific strokes or require pen-up for stroke segmentation,while tol- erating the intra-class variability caused by writing speeds,gesture sizes,writing directions,and observation ambiguity caused by viewing angles and so on in 3D space. Handwritten character recognition:In addition to inertial sensor-based approaches,many image processing techniques [3,14,16]have also been adopted for recognizing handwritten characters in a 2D plane (i.e.,image).Bahlmann et al.[4]combine DTW and SVMs to establish a Gaussian DTW(GDTW)kernel for on-line recognition of UNIPEN handwriting data.Rayar et al.[28]propose preselection method for CNN-based classification and evaluate it in handwritten character recognition in images.Rao et al.[27]propose a newly designed network structure based on an extended nonlinear kernel residual network to recognize the handwritten characters over MINIST and SVHN datasets.These approaches focus on recognizing hand-moving trajectories in a 2D plane,while our article focuses on transforming the 3D gesture into a proper 2D contour and then utilizes the contour's space-time feature to recognize contours as characters. 3 TECHNICAL CHALLENGES AND DEFINITIONS IN IN-AIR GESTURE RECOGNITION 3.1 Intra-class Variability in Sensor Data As shown in Figure 2,even when the user performs the same type of gestures (e.g.,writes"t"), the sensor data can be quite different due to the variation of writing speeds(Figure 2(a)),gesture sizes(Figure 2(b)),writing directions(Figure 2(c)),and so on.It indicates that directly using the ACM Transactions on Sensor Networks,Vol 15.No.4,Article 44.Publication date:October 2019.44:4 Y. Yin et al. and evaluation of an approach to infer eating moments using a 3-axis accelerometer in a smart￾watch. Xu et al. [35] build a classifier to identify users’ hand and finger gestures utilizing the essential features of accelerometer and gyroscope data measured from a smartwatch. Huang et al. [18] build a system to monitor brushing quality using a manual toothbrush modified by attaching small magnets to the handle and an off-the-shelf smartwatch. These approaches typically extract features from sensor data and apply machine learning techniques for gesture recognition. In-air gesture tracking: Zhou et al. [42–44] utilize a kinematic chain to track human upper￾limb motion by placing multiple devices on the arm. Cutti et al. [11] utilize the joint angles to track the movements of upper limbs by placing sensors on the chest, shoulder, arm, and wrist. Chen et al. [8] design a wearable system consisting of a pair of magnetometers on fingers and a permanent magnet affixed to the thumb and introduce uTrack to convert the thumb and fingers into a continuous input system (e.g., 3D pointing). Shen et al. [29] utilize the 5-DoF arm model and HMM to track the 3D posture of the arm, using both motion and magnetic sensors in a smartwatch. In fact, accurate in-air gesture tracking in real time can be very challenging. Besides, obtaining the 3D moving trajectory does not mean recognizing in-air gestures. In this article, we do not require accurate trajectory tracking while aiming to obtain gesture contour and recognize it as a character. Writing in the air: Zhang et al. [39] quantify data into small integral vectors based on accel￾eration orientation and then use HMM to recognize 10 Arabic numerals. Wang et al. [32] present IMUPEN to reconstruct motion trajectory and recognize handwritten digits. Bashir et al. [6] use a pen equipped with inertial sensors and apply DTW to recognize handwritten characters. Agrawal et al. [1] recognize handwritten capital letters and Arabic numerals in a 2D plane based on strokes and a grammar tree by using the built-in accelerometer in smartphone. Amma et al. [2] design a glove equipped with inertial sensors and use SVM, HMM, and statistical language model to rec￾ognize capital letters, sentences, and so on. Deselaers et al. [13] present GyroPen to reconstruct the writing path for pen-like interaction. Xu et al. [36] utilize the continuous density HMM and Viterbi algorithm to recognize handwritten digits and letters using inertial sensors. In this article, we focus on single in-air character recognition without the assistance of a language model. For a character, we do not define specific strokes or require pen-up for stroke segmentation, while tol￾erating the intra-class variability caused by writing speeds, gesture sizes, writing directions, and observation ambiguity caused by viewing angles and so on in 3D space. Handwritten character recognition: In addition to inertial sensor-based approaches, many image processing techniques [3, 14, 16] have also been adopted for recognizing handwritten characters in a 2D plane (i.e., image). Bahlmann et al. [4] combine DTW and SVMs to establish a Gaussian DTW (GDTW) kernel for on-line recognition of UNIPEN handwriting data. Rayar et al. [28] propose preselection method for CNN-based classification and evaluate it in handwritten character recognition in images. Rao et al. [27] propose a newly designed network structure based on an extended nonlinear kernel residual network to recognize the handwritten characters over MINIST and SVHN datasets. These approaches focus on recognizing hand-moving trajectories in a 2D plane, while our article focuses on transforming the 3D gesture into a proper 2D contour and then utilizes the contour’s space-time feature to recognize contours as characters. 3 TECHNICAL CHALLENGES AND DEFINITIONS IN IN-AIR GESTURE RECOGNITION 3.1 Intra-class Variability in Sensor Data As shown in Figure 2, even when the user performs the same type of gestures (e.g., writes “t”), the sensor data can be quite different due to the variation of writing speeds (Figure 2(a)), gesture sizes (Figure 2(b)), writing directions (Figure 2(c)), and so on. It indicates that directly using the ACM Transactions on Sensor Networks, Vol. 15, No. 4, Article 44. Publication date: October 2019