in their mTrack system to track the movement of a pen or a fin shift,our approach has three advantages:(1)tracking capability,(2) ger using steerable directional beams [2].Based on the fact that low latency,and (3)ability to track slow or small movements of a light reflection strength changes as a hand/finger moves,Zhang et hand/finger.We have lower latency than Doppler shift based sys- al.made customized LED/light sensors in their Okuli system to tems because Doppler shift requires Fast Fourier Transform(FFT), use visible light to track hand/finger movement [3].Based on vis- which needs to accumulate at least 2048 samples(translated to 42.7 ion processing algorithms,Leap Motion made customized infrared ms)to process,whereas we only need to accumulate 16 samples cameras to track hand/finger movements [4].Recently,Nandak- (translated to 0.3 ms).In other words,Doppler shift based systems umar et al.explored the feasibility of using commercial mobile only respond to hand/finger movement every 42.7 ms whereas our devices to track fingers/hands within a short distance.They pro- LLAP system can respond to hand/finger movement every 0.3 ms. posed fingerlO,which uses OFDM modulated sound to locate the Note that in practice,we may need to accumulate more samples fingers with accuracy of 8 mm [5]. due to the hardware limitations of mobile devices,such as 512 samples (translated to 10.7 ms)on smartphones.We can deal with 1.3 Proposed Approach slow hand/finger movement because LLAP can precisely measure In this paper,we propose a device-free gesture tracking scheme, the accumulated slow phase changes over time.We can deal with called Low-Latency Acoustic Phase (LLAP),that can be deployed small hand/finger movement because LLAP can precisely measure on existing mobile devices as a software (such as an APP)without small phase changes that is less than a full phase cycle.In contrast, any hardware modification.We use speakers and microphones Doppler-based approaches cannot detect slow or small movements that already exist on most mobile devices to perform device-free due to their limited frequency resolution,as we show in Section 3 tracking of a hand/finger.Commercial-Off-The-Shelf (COTS)mo- The second challenge is to achieve two dimensional gesture bile devices can emit and record sound waves with frequency tracking.Although LLAP can precisely measure the relative move- higher than 17 kHz,which are inaudible to most people [6].The ment distance of a hand,it cannot directly measure the absolute dis- wavelength of sound waves in this frequency range is less than 2 tance between the hand and the speaker/microphones,and therefore cm.Therefore,a small movement of a few millimeters will sig- it is hard to determine the initial hand location that is essential for nificantly change the phase of the received sound wave.Our key two dimensional tracking.To address this challenge,we use mul- idea is to use the acoustic phase to get fine-grained movement dir- tiple Continuous Waves (CW)with linearly spaced frequencies to ection and movement distance measurements.LLAP first extracts measure the path length.We observe that sound waves with dif- the sound signal reflected by the moving hand/finger after remov- ferent frequencies have different wavelengths,which leads to dif- ing the background sound signals that are relatively consistent over ferent phase shifts even if they travel through the same path.To time.Second,LLAP measures the phase changes of the sound sig- determine the path length of the reflected sound wave,we first isol- nals caused by hand/finger movements and then converts the phase ate the phase changes caused by hand/finger movement and then changes into the distance of the movement.LLAP achieves a track- apply Inverse Discrete Fourier Transform(IDFT)on the phases of ing accuracy of 3.5 mm and a latency of 15 ms on COTS mobile different sound frequencies to get the TOA of the path.By identi- phones with limited computing power.For mobile devices with two fying the TOA that has the strongest energy in the IDFT result,we or more microphones,LLAP is capable of 2-D gesture tracking that can determine the path length for the sound reflected by the mov- allows users to draw in the air with their hands/fingers. ing hand/finger.Thus,our approach can serve as a coarse-grained initial position estimation.Combining the fine-grained relative dis 1.4 Technical Challenges and Solutions tance measurement and the coarse-grained initial position estima- The first challenge is to achieve mm-level accuracy for the meas- tion,we can achieve a relatively accurate 2-D hand/finger tracking urement of hand/finger movement distance.Existing sound based 1.5 Summary of Experimental Results ranging systems either use the Time-Of-Arrival/Time-Difference- We implemented and evaluated LLAP using commercial mobile Of-Arrival(TOA/TDOA)measurements [7,8]or the Doppler shift phones without any hardware modification.Under normal indoor measurements [9.10].Traditional TOA/TDOA based systems re- noise level,for 1-D hand movement and 2-D drawing in the air, quire the device to emit bursty sound signals,such as pulses or LLAP has a tracking accuracy of 3.5 mm and 4.57 mm,respect- chirps,which are often audible to humans as these signals change ively.Under loud indoor noise level such as playing music,for 1-D abruptly [7,8].Furthermore,their distance measurement accuracy hand movement and 2-D drawing in the air,LLAP has a tracking is often in the scale of cm,except for the recent OFDM phase based accuracy of 5.81 mm and 4.89 mm,respectively.Experimental res- approach [5].Doppler shift based device-free systems do not have ults also show that LLAP can detect small hand/finger movements tracking capability and can only recognize predefined gestures be- For example,for a small single-finger movement of 5 mm,LLAP cause Doppler shift can only provide the coarse-grained measure- has a detection accuracy of 94%within a distance of 30 cm.Using ment of the speed or direction of hand/finger movements due to the gesture traces tracked by LLAP,we can recognize the characters limited frequency measurement precision [9,11,12].In contrast,to achieve mm-level hand/finger tracking accuracy,we leverage the and short words drawn in the air with an accuracy of 92.3%and 91.2%.respectively. fact that the sound reflected by a human hand is coherent to the sound emitted by the mobile device.Two signals are coherent if they have a constant phase difference and the same frequency.This 2. RELATED WORK coherency allows us to use a coherent detector to convert the re- Sound Based Localization and Tracking:TOA and TDOA ceived sound signal into a complex-valued baseband signal.Our ranging systems using sound waves has a good ranging accuracy approach is to first measure the phase change of the reflected sig- of a few centimeters because of the slower propagation speed com- nal,rather than using the noise-prone integration of the Doppler pared to radio waves [7.8,14-16].However,such systems often shift as AAMouse [13]did,and then convert the phase change to either require specially designed ultrasound transceivers [14]or the movement distance of a hand/finger.Compared with traditional emit audible probing sounds,such as short bursty sound pulses or TOA/TDOA,our approach has two advantages:(1)human inaudib- chirps [7,8,15].Furthermore,most existing sound based tracking lity,and(2)mm-level tracking accuracy.Compared with Doppler systems are not device-free as they can only track a device thatin their mTrack system to track the movement of a pen or a fin￾ger using steerable directional beams [2]. Based on the fact that light reflection strength changes as a hand/finger moves, Zhang et al. made customized LED/light sensors in their Okuli system to use visible light to track hand/finger movement [3]. Based on vis￾ion processing algorithms, Leap Motion made customized infrared cameras to track hand/finger movements [4]. Recently, Nandak￾umar et al. explored the feasibility of using commercial mobile devices to track fingers/hands within a short distance. They pro￾posed fingerIO, which uses OFDM modulated sound to locate the fingers with accuracy of 8 mm [5]. 1.3 Proposed Approach In this paper, we propose a device-free gesture tracking scheme, called Low-Latency Acoustic Phase (LLAP), that can be deployed on existing mobile devices as a software (such as an APP) without any hardware modification. We use speakers and microphones that already exist on most mobile devices to perform device-free tracking of a hand/finger. Commercial-Off-The-Shelf (COTS) mo￾bile devices can emit and record sound waves with frequency higher than 17 kHz, which are inaudible to most people [6]. The wavelength of sound waves in this frequency range is less than 2 cm. Therefore, a small movement of a few millimeters will sig￾nificantly change the phase of the received sound wave. Our key idea is to use the acoustic phase to get fine-grained movement dir￾ection and movement distance measurements. LLAP first extracts the sound signal reflected by the moving hand/finger after remov￾ing the background sound signals that are relatively consistent over time. Second, LLAP measures the phase changes of the sound sig￾nals caused by hand/finger movements and then converts the phase changes into the distance of the movement. LLAP achieves a track￾ing accuracy of 3.5 mm and a latency of 15 ms on COTS mobile phones with limited computing power. For mobile devices with two or more microphones, LLAP is capable of 2-D gesture tracking that allows users to draw in the air with their hands/fingers. 1.4 Technical Challenges and Solutions The first challenge is to achieve mm-level accuracy for the meas￾urement of hand/finger movement distance. Existing sound based ranging systems either use the Time-Of-Arrival/Time-Difference￾Of-Arrival (TOA/TDOA) measurements [7, 8] or the Doppler shift measurements [9, 10]. Traditional TOA/TDOA based systems re￾quire the device to emit bursty sound signals, such as pulses or chirps, which are often audible to humans as these signals change abruptly [7, 8]. Furthermore, their distance measurement accuracy is often in the scale of cm, except for the recent OFDM phase based approach [5]. Doppler shift based device-free systems do not have tracking capability and can only recognize predefined gestures be￾cause Doppler shift can only provide the coarse-grained measure￾ment of the speed or direction of hand/finger movements due to the limited frequency measurement precision [9, 11, 12]. In contrast, to achieve mm-level hand/finger tracking accuracy, we leverage the fact that the sound reflected by a human hand is coherent to the sound emitted by the mobile device. Two signals are coherent if they have a constant phase difference and the same frequency. This coherency allows us to use a coherent detector to convert the re￾ceived sound signal into a complex-valued baseband signal. Our approach is to first measure the phase change of the reflected sig￾nal, rather than using the noise-prone integration of the Doppler shift as AAMouse [13] did, and then convert the phase change to the movement distance of a hand/finger. Compared with traditional TOA/TDOA, our approach has two advantages: (1) human inaudib￾ility, and (2) mm-level tracking accuracy. Compared with Doppler shift, our approach has three advantages: (1) tracking capability, (2) low latency, and (3) ability to track slow or small movements of a hand/finger. We have lower latency than Doppler shift based sys￾tems because Doppler shift requires Fast Fourier Transform (FFT), which needs to accumulate at least 2048 samples (translated to 42.7 ms) to process, whereas we only need to accumulate 16 samples (translated to 0.3 ms). In other words, Doppler shift based systems only respond to hand/finger movement every 42.7 ms whereas our LLAP system can respond to hand/finger movement every 0.3 ms. Note that in practice, we may need to accumulate more samples due to the hardware limitations of mobile devices, such as 512 samples (translated to 10.7 ms) on smartphones. We can deal with slow hand/finger movement because LLAP can precisely measure the accumulated slow phase changes over time. We can deal with small hand/finger movement because LLAP can precisely measure small phase changes that is less than a full phase cycle. In contrast, Doppler-based approaches cannot detect slow or small movements due to their limited frequency resolution, as we show in Section 3. The second challenge is to achieve two dimensional gesture tracking. Although LLAP can precisely measure the relative move￾ment distance of a hand, it cannot directly measure the absolute dis￾tance between the hand and the speaker/microphones, and therefore it is hard to determine the initial hand location that is essential for two dimensional tracking. To address this challenge, we use mul￾tiple Continuous Waves (CW) with linearly spaced frequencies to measure the path length. We observe that sound waves with dif￾ferent frequencies have different wavelengths, which leads to dif￾ferent phase shifts even if they travel through the same path. To determine the path length of the reflected sound wave, we first isol￾ate the phase changes caused by hand/finger movement and then apply Inverse Discrete Fourier Transform (IDFT) on the phases of different sound frequencies to get the TOA of the path. By identi￾fying the TOA that has the strongest energy in the IDFT result, we can determine the path length for the sound reflected by the mov￾ing hand/finger. Thus, our approach can serve as a coarse-grained initial position estimation. Combining the fine-grained relative dis￾tance measurement and the coarse-grained initial position estima￾tion, we can achieve a relatively accurate 2-D hand/finger tracking. 1.5 Summary of Experimental Results We implemented and evaluated LLAP using commercial mobile phones without any hardware modification. Under normal indoor noise level, for 1-D hand movement and 2-D drawing in the air, LLAP has a tracking accuracy of 3.5 mm and 4.57 mm, respect￾ively. Under loud indoor noise level such as playing music, for 1-D hand movement and 2-D drawing in the air, LLAP has a tracking accuracy of 5.81 mm and 4.89 mm, respectively. Experimental res￾ults also show that LLAP can detect small hand/finger movements. For example, for a small single-finger movement of 5 mm, LLAP has a detection accuracy of 94% within a distance of 30 cm. Using gesture traces tracked by LLAP, we can recognize the characters and short words drawn in the air with an accuracy of 92.3% and 91.2%, respectively. 2. RELATED WORK Sound Based Localization and Tracking: TOA and TDOA ranging systems using sound waves has a good ranging accuracy of a few centimeters because of the slower propagation speed com￾pared to radio waves [7, 8, 14–16]. However, such systems often either require specially designed ultrasound transceivers [14] or emit audible probing sounds, such as short bursty sound pulses or chirps [7, 8, 15]. Furthermore, most existing sound based tracking systems are not device-free as they can only track a device that