Gait Recognition Using WiFi Signals Wei Wang Alex X.Liu Muhammad Shahzads State Key Laboratory for Novel Software Technology,Nanjing University,China Department of Computer Science and Engineering,Michigan State University,USA s Department of Computer Science,North Carolina State University,USA Email:ww@cs.nju.edu.cn,alexliu @cse.msu.edu,mshahza@ncsu.edu Abstract In this paper,we propose WifiU,which uses commercial WiFi devices to capture fine-grained gait patterns to recognize hu- mans.The intuition is that due to the differences in gaits of different people,the WiFi signal reflected by a walking hu- man generates unique variations in the Channel State Inform- ation(CSD)on the WiFi receiver.To profile human movement using CSI,we use signal processing techniques to generate spectrograms from CSI measurements so that the resulting 7.7m spectrograms are similar to those generated by specifically (a)Application scenario (b)Data collection environment designed Doppler radars.To extract features from spectro- grams that best characterize the walking pattern,we perform Figure 1.WifiU system overview autocorrelation on the torso reflection to remove imperfection in spectrograms.We evaluated WifiU on a dataset with 2,800 be a laptop.Figure 1 shows an overview of our WifiU system. gait instances collected from 50 human subjects walking in a In WifiU,the specific information that the receiver measures room with an area of 50 square meters.Experimental results is the Channel State Information(CSD)of each received WiFi show that WifU achieves top-1,top-2,and top-3 recognition frame.With a human walking around.as a human is mostly accuracies of 79.28%,89.52%,and 93.05%,respectively. made of water,the WiFi signal reflected by the human body generates unique,although small,variations in the CSI meas- ACM Classification Keywords urements on the receiver due to the well-known multi-path H.1.2 User/Machine Systems;I.5.Pattern Recognition effect of wireless signals.These variations in CSI allow us to use signal processing techniques to obtain gait information Author Keywords such as walking speed,gait cycle time,footstep length,and Gait Recognition;Device-free Sensing. movement speeds of legs and torso.As it is well known that humans have quite unique gait [8,15].the gait patterns that INTRODUCTION the WiFi receiver obtains can be used to recognize the walk- ing human subject.Fundamentally,WifiU recognizes humans Motivation and Proposed Approach based on who they are because WifiU extracts unique bio- With the success of WiFi industry,commercially available metrics information from WiFi signals and uses it to perform WiFi devices can not only achieve high speed and low cost, human recognition. but can also measure small changes in WiFi signals and thus sense the changes in their surrounding environments caused WifiU enables many potential applications that require user by moving objects such as humans [30,31].In this paper, identification.For example,a smart building can recognize we propose WifiU,which uses Commercial Off-The-Shelf the user using WifiU,while the user is walking along the cor- (COTS)WiFi devices to capture fine-grained gait patterns so ridor.Based on the user identity,it can automatically open the that we can recognize humans.WifiU consists of two WiFi door when he/she approaches his/her office.Similarly,smart devices,one for continuously sending signals,which can be a home applications can use WifiU to adjust background music router,and one for continuously receiving signals,which can or ambient temperature based on who is at home. Compared with traditional gait recognition systems,which Permission to make digital or hard copies of all or part of this work for personal or use cameras [17,floor sensors [18],or wearable sensors [8 to capture gait information,WifiU is easier to deploy and has tion on the first page.Copyrights for components of this work owned by others than better coverages.From the deployment perspective,WifiU ACM must be honored.Abstracting with credit is permitted.To copy otherwise,or re publish,to post on servers or to redistribute to lists,requires prior specific permission does not require any special hardware(such as floor sensors) and/or a fee.Request permissions from permissions@acm.org. and the human subject does not require to wear any hardware UbiComp '16,September 12-16.2016.Heidelberg.Germany (such as accelerometers).WiFi devices are ubiquitous and ©2016AC3M.ISBN978-1-4503-4461-6/1609.s15.00 most homes/offices are covered by WiFi signals.The hard- D0L:http:/dx.doi.org/10.1145/2971648.2971670
Gait Recognition Using WiFi Signals Wei Wang† Alex X. Liu†‡ Muhammad Shahzad§ †State Key Laboratory for Novel Software Technology, Nanjing University, China ‡Department of Computer Science and Engineering, Michigan State University, USA §Department of Computer Science, North Carolina State University, USA Email: ww@cs.nju.edu.cn, alexliu@cse.msu.edu, mshahza@ncsu.edu Abstract In this paper, we propose WifiU, which uses commercial WiFi devices to capture fine-grained gait patterns to recognize humans. The intuition is that due to the differences in gaits of different people, the WiFi signal reflected by a walking human generates unique variations in the Channel State Information (CSI) on the WiFi receiver. To profile human movement using CSI, we use signal processing techniques to generate spectrograms from CSI measurements so that the resulting spectrograms are similar to those generated by specifically designed Doppler radars. To extract features from spectrograms that best characterize the walking pattern, we perform autocorrelation on the torso reflection to remove imperfection in spectrograms. We evaluated WifiU on a dataset with 2,800 gait instances collected from 50 human subjects walking in a room with an area of 50 square meters. Experimental results show that WifiU achieves top-1, top-2, and top-3 recognition accuracies of 79.28%, 89.52%, and 93.05%, respectively. ACM Classification Keywords H.1.2 User/Machine Systems; I.5. Pattern Recognition Author Keywords Gait Recognition; Device-free Sensing. INTRODUCTION Motivation and Proposed Approach With the success of WiFi industry, commercially available WiFi devices can not only achieve high speed and low cost, but can also measure small changes in WiFi signals and thus sense the changes in their surrounding environments caused by moving objects such as humans [30, 31]. In this paper, we propose WifiU, which uses Commercial Off-The-Shelf (COTS) WiFi devices to capture fine-grained gait patterns so that we can recognize humans. WifiU consists of two WiFi devices, one for continuously sending signals, which can be a router, and one for continuously receiving signals, which can Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. UbiComp ’16, September 12-16, 2016, Heidelberg, Germany © 2016 ACM. ISBN 978-1-4503-4461-6/16/09. . . $15.00 DOI: http://dx.doi.org/10.1145/2971648.2971670 Signal analysis WiFi router WiFi signal reflection Laptop WiFi signal (a) Application scenario Table Table Walking route (5.5m) 7.7 m 6.5 m 1.6 m sender receiver (b) Data collection environment Figure 1. WifiU system overview be a laptop. Figure 1 shows an overview of our WifiU system. In WifiU, the specific information that the receiver measures is the Channel State Information (CSI) of each received WiFi frame. With a human walking around, as a human is mostly made of water, the WiFi signal reflected by the human body generates unique, although small, variations in the CSI measurements on the receiver due to the well-known multi-path effect of wireless signals. These variations in CSI allow us to use signal processing techniques to obtain gait information such as walking speed, gait cycle time, footstep length, and movement speeds of legs and torso. As it is well known that humans have quite unique gait [8, 15], the gait patterns that the WiFi receiver obtains can be used to recognize the walking human subject. Fundamentally, WifiU recognizes humans based on who they are because WifiU extracts unique biometrics information from WiFi signals and uses it to perform human recognition. WifiU enables many potential applications that require user identification. For example, a smart building can recognize the user using WifiU, while the user is walking along the corridor. Based on the user identity, it can automatically open the door when he/she approaches his/her office. Similarly, smart home applications can use WifiU to adjust background music or ambient temperature based on who is at home. Compared with traditional gait recognition systems, which use cameras [17], floor sensors [18], or wearable sensors [8] to capture gait information, WifiU is easier to deploy and has better coverages. From the deployment perspective, WifiU does not require any special hardware (such as floor sensors) and the human subject does not require to wear any hardware (such as accelerometers). WiFi devices are ubiquitous and most homes/offices are covered by WiFi signals. The hard-
ware that we experimented with in this work,namely Net- the scenario where a burglar attempts to know who is at home Gear JR6100 WiFi router and ThinkPad X200 laptop (with by eavesdropping the WiFi signal emitted by the WiFi router Intel 5300 WiFi NIC),are COTS devices and we do not re- in the victim's house.As WiFi signals can penetrate through quire any hardware modification to these devices.Further- obstacles such as furniture,wooden doors,and walls [3].the more,unlike cameras,WifiU does not require lighting and burglar only needs to passively measure the CSI of the signal works in dark just as well as in light outside the house without decoding the content of the WiFi packets.Therefore,it is hard for the victim to prevent privacy Technical Challenges and Solutions breaches.While protecting privacy breach is not the focus of The first challenge is to profile gait patterns using CSI dy- this paper,we hope that this work will bring this privacy issue namics.Extracting gait information from CSI signals is dif- to the research community and inspire future work. ficult because the signal reflections of different body parts are mixed together in the CSI waveform.As different hu- RELATED WORK man body parts move at different speeds while walking,the Gait Based Human Recognition radio signal reflections from different body parts have differ- Prior gait based human recognition schemes mostly use in- ent frequencies.To separate the radio signal reflections from formation collected from the following three types of sensors: different body parts,we convert the CSI waveforms (of two cameras.floor sensors.and wearable sensors [81.Camera dimensions:time and amplitude)to spectrograms in the time- based schemes first generate a sequence of human silhouettes frequency domain (of three dimensions:time,frequency,and from the video frames by separating the moving objects from amplitude).We apply spectrogram enhancement techniques the background,and then use the sequence of human silhou- to reduce signal noises.The resulting spectrograms give us ettes to perform recognition [17].An example camera based detailed human gait information similar to those generated scheme achieves an accuracy of 91%on the USF HumanID by Doppler radars [22.271. Database with 122 subjects [291.Floor sensor based schemes The second challenge is to extract features from spectrograms use force sensors under the floor to detect human footsteps that best characterize the walking pattern of a human.There [18.An example floor sensor based scheme achieves an ac- are various metrics (such as walking speed and gait cycle curacy of 90%~97.5%for identification a single target per- time)that can be used to characterize human walking pat- son [281.Wearable sensor based schemes use the acceler- terns.However,obtaining these measurements from a spec- ometers equipped in smart phones and wearable devices to trogram is challenging because we need to precisely measure collect gait information and extract features for human re- the time of gait cycles.For example,to measure gait cycle cognition [6,19.An example wearable sensor based scheme time,i.e.,the time between two consecutive instances for the achieves an accuracy of 90%over 6 human subjects 24. right heel hits the ground [33],we need to determine the time that the right heel hitting the ground,which is non-trivial.To Activity and Gait Recognition using Radar Signals As a human body can reflect wireless signals,human move- accurately measure gait cycle time,we perform autocorrela- ment can be analyzed using the Doppler shift of the re- tion on the contours of the torso reflection to remove small imperfection in spectrograms.We also generate a spectro- flected signal or the time-of-flight measurements obtained gram signature to distinguish persons with similar walking through Frequency-Modulated Continuous-Wave (FMCW) radars.The micro-Doppler signature,which describes the ve- speeds and footstep lengths. locity of different body parts,can be used for human activity Summary of Experimental Results recognition [16,21,22].Using the Boulic walking model,it is We implemented WifiU on COTS laptop and WiFi router. possible to estimate parameters from the micro-Doppler sig- We conducted experiments on our gait database that contains nature and use these parameters to rebuild the animation of more than 2,800 gait instances collected from 50 human sub- the walking process [27].Tahmoush et al.built a radar based jects walking in a typical laboratory with an area of 50 square gait recognition system that uses the stride rate as features meters as shown in Figure 1(b).We anonymized all collected and achieve recognition accuracy of 80%over 8 human sub- data to protect privacy.Over the 50 subjects,WifiU achieves jects [25].Adib et al.used FMCW Radar to capture a human recognition accuracies of 79.28%,89.52%,and 93.05%for figure behind the wall and to identify the user with an accur- top-1,top-2,and top-3 candidates,respectively acy of 88%over 15 human subjects [1].Compared to radar signals,WiFi signals are much easier and cheaper to obtain, Limitations and Privacy Implications but have much narrower bandwidth(20 MHz compared to The current implementation of WifiU has two limitations. 1.79 GHz of the FMCW radar used in WiTrack [21)and thus First,the recognition system is suitable only for confined have much lower time-resolutions.This makes human gait spaces,such as a corridor or a narrow entrance,because the recognition using WiFi signals much harder than using radar users must walk on a predefined path in a predefined direc- signals.Consequently,the techniques in this paper are funda- tion.Second,WifiU can extract the gait pattern only when mentally different from those used in FMCW radars there is a single user walking on the predefined path.Our future work is to explore the possibility of using multiple Activity Recognition using WiFi Signals devices to separate gait signals from multiple users. CSI measurements have been used for applications such as human activity recognition [31,32],fall detection [12],and From the perspective of privacy,hackers can potentially use even object localization [23].E-eyes utilized CSI histogram WifiU to identify"victims"without being detected.Consider to achieve in-place and walking activity recognition with an
ware that we experimented with in this work, namely NetGear JR6100 WiFi router and ThinkPad X200 laptop (with Intel 5300 WiFi NIC), are COTS devices and we do not require any hardware modification to these devices. Furthermore, unlike cameras, WifiU does not require lighting and works in dark just as well as in light. Technical Challenges and Solutions The first challenge is to profile gait patterns using CSI dynamics. Extracting gait information from CSI signals is dif- ficult because the signal reflections of different body parts are mixed together in the CSI waveform. As different human body parts move at different speeds while walking, the radio signal reflections from different body parts have different frequencies. To separate the radio signal reflections from different body parts, we convert the CSI waveforms (of two dimensions: time and amplitude) to spectrograms in the timefrequency domain (of three dimensions: time, frequency, and amplitude). We apply spectrogram enhancement techniques to reduce signal noises. The resulting spectrograms give us detailed human gait information similar to those generated by Doppler radars [22, 27]. The second challenge is to extract features from spectrograms that best characterize the walking pattern of a human. There are various metrics (such as walking speed and gait cycle time) that can be used to characterize human walking patterns. However, obtaining these measurements from a spectrogram is challenging because we need to precisely measure the time of gait cycles. For example, to measure gait cycle time, i.e., the time between two consecutive instances for the right heel hits the ground [33], we need to determine the time that the right heel hitting the ground, which is non-trivial. To accurately measure gait cycle time, we perform autocorrelation on the contours of the torso reflection to remove small imperfection in spectrograms. We also generate a spectrogram signature to distinguish persons with similar walking speeds and footstep lengths. Summary of Experimental Results We implemented WifiU on COTS laptop and WiFi router. We conducted experiments on our gait database that contains more than 2,800 gait instances collected from 50 human subjects walking in a typical laboratory with an area of 50 square meters as shown in Figure 1(b). We anonymized all collected data to protect privacy. Over the 50 subjects, WifiU achieves recognition accuracies of 79.28%, 89.52%, and 93.05% for top-1, top-2, and top-3 candidates, respectively. Limitations and Privacy Implications The current implementation of WifiU has two limitations. First, the recognition system is suitable only for confined spaces, such as a corridor or a narrow entrance, because the users must walk on a predefined path in a predefined direction. Second, WifiU can extract the gait pattern only when there is a single user walking on the predefined path. Our future work is to explore the possibility of using multiple devices to separate gait signals from multiple users. From the perspective of privacy, hackers can potentially use WifiU to identify “victims” without being detected. Consider the scenario where a burglar attempts to know who is at home by eavesdropping the WiFi signal emitted by the WiFi router in the victim’s house. As WiFi signals can penetrate through obstacles such as furniture, wooden doors, and walls [3], the burglar only needs to passively measure the CSI of the signal outside the house without decoding the content of the WiFi packets. Therefore, it is hard for the victim to prevent privacy breaches. While protecting privacy breach is not the focus of this paper, we hope that this work will bring this privacy issue to the research community and inspire future work. RELATED WORK Gait Based Human Recognition Prior gait based human recognition schemes mostly use information collected from the following three types of sensors: cameras, floor sensors, and wearable sensors [8]. Camera based schemes first generate a sequence of human silhouettes from the video frames by separating the moving objects from the background, and then use the sequence of human silhouettes to perform recognition [17]. An example camera based scheme achieves an accuracy of 91% on the USF HumanID Database with 122 subjects [29]. Floor sensor based schemes use force sensors under the floor to detect human footsteps [18]. An example floor sensor based scheme achieves an accuracy of 90%∼97.5% for identification a single target person [28]. Wearable sensor based schemes use the accelerometers equipped in smart phones and wearable devices to collect gait information and extract features for human recognition [6, 19]. An example wearable sensor based scheme achieves an accuracy of 90% over 6 human subjects [24]. Activity and Gait Recognition using Radar Signals As a human body can reflect wireless signals, human movement can be analyzed using the Doppler shift of the re- flected signal or the time-of-flight measurements obtained through Frequency-Modulated Continuous-Wave (FMCW) radars. The micro-Doppler signature, which describes the velocity of different body parts, can be used for human activity recognition [16,21,22]. Using the Boulic walking model, it is possible to estimate parameters from the micro-Doppler signature and use these parameters to rebuild the animation of the walking process [27]. Tahmoush et al. built a radar based gait recognition system that uses the stride rate as features and achieve recognition accuracy of 80% over 8 human subjects [25]. Adib et al. used FMCW Radar to capture a human figure behind the wall and to identify the user with an accuracy of 88% over 15 human subjects [1]. Compared to radar signals, WiFi signals are much easier and cheaper to obtain, but have much narrower bandwidth (20 MHz compared to 1.79 GHz of the FMCW radar used in WiTrack [2]) and thus have much lower time-resolutions. This makes human gait recognition using WiFi signals much harder than using radar signals. Consequently, the techniques in this paper are fundamentally different from those used in FMCW radars. Activity Recognition using WiFi Signals CSI measurements have been used for applications such as human activity recognition [31, 32], fall detection [12], and even object localization [23]. E-eyes utilized CSI histogram to achieve in-place and walking activity recognition with an
accuracy of 92%[32].WiFall used anomaly detection al- there is no moving object around,the magnitude of CSI re- gorithm on CSI values to detect falling [12].PinLoc gathered mains relatively constant.However,a moving human can location dependent CSI value distributions to localize a tar- significantly distort the CSI magnitude because human bodies get with accuracy of several centimeters [23].MRW used are good reflectors of wireless signals.When a human moves. CSI or Received Signal Strength (RSS)values to monitor the wireless signal reflected by his body will go through a dif- the presence of human movement behind a concrete wall [4] ferent path with different length,as illustrated in Figure 1(a). CARM used CSI-speed model to extract human movement speeds from CSI signal [31].However,these speed features Governed by the principle of superposition of waves,signal are still not accurate enough for gait pattern recognition.In reflected by human may add constructively or destructively comparison,we perform time-frequency analysis on CSI val- with WiFi signals traveled through other paths,e.g.,the Line- ues to obtain spectrograms,and then to measure human walk- Of-Sight (LOS)path.Whether these WiFi signals are added ing metrics,such as gait cycle time,torso and leg speed,and constructively or destructively is determined by the relative footstep length from the spectrograms.Note that there are phase differences between these signals [26].The phase of schemes that use special hardware(such as USRP [20]and the signal changes by 2 when the length change of the sig- directional antennas [13,30])to get fine-grained measure- nal path is equal to the signal wavelength.Whenever the hu- ments from WiFi signals:in comparison,we only use COTS man moves by half of the wavelength,the path length of the WiFi devices. human reflected signal will change by the amount of the sig- nal wavelength.Consequently,we will observe a full cycle of In recent work parallel with ours,Zeng et al.built a system magnitude changes in CSI values according to the principle called WiWho that also uses WiFi CSI signals to recognize of superposition of waves.The factor of 2 in the path length human gaits [34].In WiWho,the human subject was asked to change is due to the round trip path travelled by the reflected walk on a path with a distance of I meter parallel to the Line- WiFi signal.The signal wavelength A for 5 GHz WiFi signal Of-Sight (LOS)path between the WiFi sender and receiver. band is 5.15~5.79 cm.This implies that we can detect small The recognition accuracy of WiWho is 92%to 80%for 2 to 6 movements of a few centimeters by observing the magnitude human subjects.Compared to WiWho,WifiU uses advanced changes of CSI values.For a walking human who moves at signal processing algorithms to extract CSI variations so that the speed of 1 meter per second,we observe 34~38 cycles of it can recognize gaits at a distance of more than 6 meters to magnitude fluctuations in CSI values,given the wavelength of the LOS path.Moreover,WifiU extracts more elaborated gait 5.15~5.79 cm,as each cycle represents the person moves by features from CSI signals the distance of half-wavelength.This provides a very detailed WIFI SIGNAL PROCESSING measurement of the human walking speed. In this section,we first collect CSI measurements from WiFi However,the CSI measurements obtained from commercial signals using COTS WiFi devices.Second,we use the Prin- WiFi cards contain noises from various sources such as in- cipal Component Analysis (PCA)technique to extract the terference coming from nearby devices,transmission power principal components from the correlated CSI measurements adaptation at the sender,and imperfect clock synchronization so that the uncorrelated noises in different subcarriers are re- [101.Figure 2(a)shows the magnitude of a raw CSI stream duced.Third,we use Short Time Fourier Transform(STFT) (Stream A)captured while a human is walking around.Al- to convert PCA components into spectrograms. Fourth though we can observe the fluctuations in CSI values caused we apply frequency domain denoising algorithms(such as by the moving human subject,these fluctuations are irregu- noise floor subtraction,spectrogram superimposition,and 2- lar due to environmental noises.Thus,we must denoise CSI dimensional filtering)to further enhance the spectrogram. measurements before we extract human gait information. CSI Data Collection WifiU collects CSI measurements on the receiving end of a Existing CSI denoising schemes,such as low-pass filters [30] WiFi link between two WiFi devices.For each pair of a send- do not work well for our purpose because CSI streams con- ing antenna and a receiving antenna,we obtain CSI values tain high-level impulse and burst noises.Figure 2(a)shows an from 30 OFDM subcarriers used by 802.11n [14].Thus,we example CSI stream with impulse noises at the time of 10.55 get 2 x 3 x 30=180 CSI values for each received 802.11n seconds as pointed by an arrow.Figure 2(b)shows the low- frame when the sender has 2 antennas and the receiver has pass filtering result of CSI Stream A after passing through 3 antennas.The sequence of CSI values for each subcar- a Butterworth filter with a cutoff frequency at 150 Hz.We rier for a given pair of sending/receiving antenna is called can still observe small residual fluctuations,as pointed by a CSI stream.As our system sends 2.500 WiFi frames per an arrow,after low-pass filtering,due to the wide bandwidth second,we collected 2,500 CSI values for each of the 180 of impulse noises.Figure 2(b)plots another CSI Stream B. CSI streams in one second.We removed the impact of Car- which is measured on the same sender/receiver antenna pair rier Frequency Offset(CFO)by using only the amplitude of but with a subcarrier frequency about 10 MHz higher than the CSI values while ignoring the CSI phase,as described in that of CSI Stream A.Zooming into the waveform segments our earlier work [31. separated by the four vertical reference lines,we observe that the "valleys"for Stream B always appear earlier than that of Denoising CSI Measurements Stream A.This indicates that variations in CSI streams have The CSI values describe how the phase and magnitude of the different phases.Different CSI streams often have differ- wireless signal change when the signal travels from the send- ent phases because they differ in their subcarrier frequency ing antenna to the receiving antenna over a subcarrier.When
accuracy of 92% [32]. WiFall used anomaly detection algorithm on CSI values to detect falling [12]. PinLoc gathered location dependent CSI value distributions to localize a target with accuracy of several centimeters [23]. MRW used CSI or Received Signal Strength (RSS) values to monitor the presence of human movement behind a concrete wall [4]. CARM used CSI-speed model to extract human movement speeds from CSI signal [31]. However, these speed features are still not accurate enough for gait pattern recognition. In comparison, we perform time-frequency analysis on CSI values to obtain spectrograms, and then to measure human walking metrics, such as gait cycle time, torso and leg speed, and footstep length from the spectrograms. Note that there are schemes that use special hardware (such as USRP [20] and directional antennas [13, 30]) to get fine-grained measurements from WiFi signals; in comparison, we only use COTS WiFi devices. In recent work parallel with ours, Zeng et al. built a system called WiWho that also uses WiFi CSI signals to recognize human gaits [34]. In WiWho, the human subject was asked to walk on a path with a distance of 1 meter parallel to the LineOf-Sight (LOS) path between the WiFi sender and receiver. The recognition accuracy of WiWho is 92% to 80% for 2 to 6 human subjects. Compared to WiWho, WifiU uses advanced signal processing algorithms to extract CSI variations so that it can recognize gaits at a distance of more than 6 meters to the LOS path. Moreover, WifiU extracts more elaborated gait features from CSI signals. WIFI SIGNAL PROCESSING In this section, we first collect CSI measurements from WiFi signals using COTS WiFi devices. Second, we use the Principal Component Analysis (PCA) technique to extract the principal components from the correlated CSI measurements so that the uncorrelated noises in different subcarriers are reduced. Third, we use Short Time Fourier Transform (STFT) to convert PCA components into spectrograms. Fourth, we apply frequency domain denoising algorithms (such as noise floor subtraction, spectrogram superimposition, and 2- dimensional filtering) to further enhance the spectrogram. CSI Data Collection WifiU collects CSI measurements on the receiving end of a WiFi link between two WiFi devices. For each pair of a sending antenna and a receiving antenna, we obtain CSI values from 30 OFDM subcarriers used by 802.11n [14]. Thus, we get 2 × 3 × 30 = 180 CSI values for each received 802.11n frame when the sender has 2 antennas and the receiver has 3 antennas. The sequence of CSI values for each subcarrier for a given pair of sending/receiving antenna is called a CSI stream. As our system sends 2,500 WiFi frames per second, we collected 2,500 CSI values for each of the 180 CSI streams in one second. We removed the impact of Carrier Frequency Offset (CFO) by using only the amplitude of the CSI values while ignoring the CSI phase, as described in our earlier work [31]. Denoising CSI Measurements The CSI values describe how the phase and magnitude of the wireless signal change when the signal travels from the sending antenna to the receiving antenna over a subcarrier. When there is no moving object around, the magnitude of CSI remains relatively constant. However, a moving human can significantly distort the CSI magnitude because human bodies are good reflectors of wireless signals. When a human moves, the wireless signal reflected by his body will go through a different path with different length, as illustrated in Figure 1(a). Governed by the principle of superposition of waves, signal reflected by human may add constructively or destructively with WiFi signals traveled through other paths, e.g., the LineOf-Sight (LOS) path. Whether these WiFi signals are added constructively or destructively is determined by the relative phase differences between these signals [26]. The phase of the signal changes by 2π when the length change of the signal path is equal to the signal wavelength. Whenever the human moves by half of the wavelength, the path length of the human reflected signal will change by the amount of the signal wavelength. Consequently, we will observe a full cycle of magnitude changes in CSI values according to the principle of superposition of waves. The factor of 2 in the path length change is due to the round trip path travelled by the reflected WiFi signal. The signal wavelength λ for 5 GHz WiFi signal band is 5.15∼5.79 cm. This implies that we can detect small movements of a few centimeters by observing the magnitude changes of CSI values. For a walking human who moves at the speed of 1 meter per second, we observe 34∼38 cycles of magnitude fluctuations in CSI values, given the wavelength of 5.15∼5.79 cm, as each cycle represents the person moves by the distance of half-wavelength. This provides a very detailed measurement of the human walking speed. However, the CSI measurements obtained from commercial WiFi cards contain noises from various sources such as interference coming from nearby devices, transmission power adaptation at the sender, and imperfect clock synchronization [10]. Figure 2(a) shows the magnitude of a raw CSI stream (Stream A) captured while a human is walking around. Although we can observe the fluctuations in CSI values caused by the moving human subject, these fluctuations are irregular due to environmental noises. Thus, we must denoise CSI measurements before we extract human gait information. Existing CSI denoising schemes, such as low-pass filters [30], do not work well for our purpose because CSI streams contain high-level impulse and burst noises. Figure 2(a) shows an example CSI stream with impulse noises at the time of 10.55 seconds as pointed by an arrow. Figure 2(b) shows the lowpass filtering result of CSI Stream A after passing through a Butterworth filter with a cutoff frequency at 150 Hz. We can still observe small residual fluctuations, as pointed by an arrow, after low-pass filtering, due to the wide bandwidth of impulse noises. Figure 2(b) plots another CSI Stream B, which is measured on the same sender/receiver antenna pair but with a subcarrier frequency about 10 MHz higher than that of CSI Stream A. Zooming into the waveform segments separated by the four vertical reference lines, we observe that the “valleys” for Stream B always appear earlier than that of Stream A. This indicates that variations in CSI streams have different phases. Different CSI streams often have different phases because they differ in their subcarrier frequency
Stream A 30 -Stream B 10.1 10.2 10.5 10.6 10.7 20 10.1 10.2 10.5 10.6 107 10.1 10.9 10.3 10.5 10.6 (a)Raw CSI waveforms(Stream A) (b)CSI waveforms after low-pass filtering (c)CSI waveforms after PCA denoising Figure 2.CSI waveforms for a person walking along a straight path as in Figure 1(b) and thus their radio wavelengths also differ.The phases of moving speeds,we convert the PCA denoised CSI waveforms these CSI streams can differ by a and their waveforms may into the time-frequency domain.We use the Short-Time Four- even cancel each other.Therefore,directly adding these CSI ier Transform(STFT)technique to transform the waveforms streams is also not a good choice for noise removal. to spectrograms so that CSI waveforms can be analyzed in the We use Principal Component Analysis(PCA)based denois- time-frequency domain.The spectrogram uses a sliding win- dow to cut a small piece of samples from the waveform and ing algorithm to remove noises in the CSI signal [311.Phase differences in CSI streams can be characterized by their cross then performs Fast Fourier transform(FFT)on the waveform. correlations.PCA can automatically discover the correlation The spectrogram has three dimensions:time,frequency,and between CSI streams and recombine CSI streams to extract FFT amplitude.The window size for FFT determines the components that represent the variation caused by human tradeoffs between frequency and time resolution of STFT. activity.To remove the noise,we extract the first twenty PCA With a larger window size,the STFT has higher frequency resolution but lower time resolutions.CSI measurements for components from the 180 CSI streams and discard the rest, human walking have frequency of 30~40 Hz and changes in which are mostly noisy components.The optimal number of tens of milliseconds.Thus,we choose to use an FFT size PCA components is determined by experiments shown in the experimental results section.The PCA denoised CSI wave- of 1024 samples and sliding window step size of 32 samples forms are much smoother than those generated by low-pass because it gives suitable frequency resolution of 2.44 Hz and filtering and preserve the details in CSI amplitude changes. time resolution of 12.8 ms to trace the changes in walking signals.Figure 3 shows an example spectrogram in a heat Figure 2(c)shows the second PCA component derived from the raw CSI streams in Figure 2(a).We observe that there map,where hotter color represents higher FFT amplitude.To are 17 peaks for the 0.5 second between 10~10.5 seconds build the heat map for the spectrogram,we use the first 80 in Figure 2(c).This gives 34 cycles/sec fluctuations in CSI. FFT magnitude of each chunk to represent the energy in the matching the walk speed of 1m/s of the human subject. frequency range of 0~146 Hz Spectrogram Enhancement Spectrogram Generation After a spectrogram is generated,we apply spectrogram en- It is still difficult to extract human gait information from a hancement techniques to further reduce the noise in the spec- PCA denoised CSI waveform because the signal reflections trogram. First,we get the energy level for each chunk by of different body parts are mixed together in the waveform. summing the magnitude of the first 80 FFT points.We Different human body parts move at different speeds during ignore chunks with energy level lower than an empirical different phases in walking.A gait cycle of human walking threshold so that silent periods are removed.We normalize has two phases:the stance phase and the swing phase [33]. the FFT magnitudes for each remaining chunk by dividing The stance phase starts at the time that the right heel touches them with the energy level of the chunk.Second,we ap- the ground.During the stance phase,the left leg moves for- ply frequency domain denoising method by subtracting the ward.At the end of the stance phase.the right toe leaves the noise floor from the spectrogram [7].The noise floor level is ground.In the swing phase,the human body is supported by estimated through a long-term mean over the spectrograms, the left leg and the right leg is swinging forward.Thus,the e.g.,average magnitude over 4 seconds.If the resulting mag- right leg moves at a higher speed than the torso and the left leg nitude becomes negative after subtraction,we set it to zero. is almost static during this phase [27].As different body parts Third,we superimpose the spectrograms generated by the move at different speeds,the radio signal reflections of differ- first twenty PCA results by adding up the magnitude of cor- ent body parts have different frequencies.CSI frequency is responding time-frequency blocks.Fourth,we apply a 2- determined by human moving speeds as f=2v/A,where v dimensional Gaussian low-pass filter,with a size of 5 and is the moving speed,f is the frequency of the CSI waveform, o of 0.8,on the resulting spectrogram.Finally,we obtain and A is the radio signal wavelength.For example,a moving a high quality spectrogram that gives detailed information for speed of 1 m/s can be converted to the frequency of 34.54 Hz the walking process of a human subject. when the radio wavelength A is 5.79cm. FEATURE EXTRACTION As the radio signal reflections of different body parts have dif- In this section,we present how gait features can be extracted ferent frequencies,the CSI fluctuations of different frequen- from the CSI spectrogram.The gait features include walking cies are separable in the frequency domain.Thus,to separate speed,gait cycle time,footstep length,movement speeds of the radio signal reflections of different body parts of different torso and legs,and spectrogram signatures
10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 35 40 45 50 55 Time (seconds) CSI amplitude Impulse noises (a) Raw CSI waveforms (Stream A) 10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 25 30 35 40 45 50 55 Time (seconds) CSI amplitude Stream A Stream B Residual noises (b) CSI waveforms after low-pass filtering 10 10.1 10.2 10.3 10.4 10.5 10.6 10.7 −30 −20 −10 0 10 20 30 Time (seconds) PCA amplitude (c) CSI waveforms after PCA denoising Figure 2. CSI waveforms for a person walking along a straight path as in Figure 1(b) and thus their radio wavelengths also differ. The phases of these CSI streams can differ by π and their waveforms may even cancel each other. Therefore, directly adding these CSI streams is also not a good choice for noise removal. We use Principal Component Analysis (PCA) based denoising algorithm to remove noises in the CSI signal [31]. Phase differences in CSI streams can be characterized by their cross correlations. PCA can automatically discover the correlation between CSI streams and recombine CSI streams to extract components that represent the variation caused by human activity. To remove the noise, we extract the first twenty PCA components from the 180 CSI streams and discard the rest, which are mostly noisy components. The optimal number of PCA components is determined by experiments shown in the experimental results section. The PCA denoised CSI waveforms are much smoother than those generated by low-pass filtering and preserve the details in CSI amplitude changes. Figure 2(c) shows the second PCA component derived from the raw CSI streams in Figure 2(a). We observe that there are 17 peaks for the 0.5 second between 10∼10.5 seconds in Figure 2(c). This gives 34 cycles/sec fluctuations in CSI, matching the walk speed of 1m/s of the human subject. Spectrogram Generation It is still difficult to extract human gait information from a PCA denoised CSI waveform because the signal reflections of different body parts are mixed together in the waveform. Different human body parts move at different speeds during different phases in walking. A gait cycle of human walking has two phases: the stance phase and the swing phase [33]. The stance phase starts at the time that the right heel touches the ground. During the stance phase, the left leg moves forward. At the end of the stance phase, the right toe leaves the ground. In the swing phase, the human body is supported by the left leg and the right leg is swinging forward. Thus, the right leg moves at a higher speed than the torso and the left leg is almost static during this phase [27]. As different body parts move at different speeds, the radio signal reflections of different body parts have different frequencies. CSI frequency is determined by human moving speeds as f = 2v/λ, where v is the moving speed, f is the frequency of the CSI waveform, and λ is the radio signal wavelength. For example, a moving speed of 1 m/s can be converted to the frequency of 34.54 Hz when the radio wavelength λ is 5.79cm. As the radio signal reflections of different body parts have different frequencies, the CSI fluctuations of different frequencies are separable in the frequency domain. Thus, to separate the radio signal reflections of different body parts of different moving speeds, we convert the PCA denoised CSI waveforms into the time-frequency domain. We use the Short-Time Fourier Transform (STFT) technique to transform the waveforms to spectrograms so that CSI waveforms can be analyzed in the time-frequency domain. The spectrogram uses a sliding window to cut a small piece of samples from the waveform and then performs Fast Fourier transform (FFT) on the waveform. The spectrogram has three dimensions: time, frequency, and FFT amplitude. The window size for FFT determines the tradeoffs between frequency and time resolution of STFT. With a larger window size, the STFT has higher frequency resolution but lower time resolutions. CSI measurements for human walking have frequency of 30∼40 Hz and changes in tens of milliseconds. Thus, we choose to use an FFT size of 1024 samples and sliding window step size of 32 samples because it gives suitable frequency resolution of 2.44 Hz and time resolution of 12.8 ms to trace the changes in walking signals. Figure 3 shows an example spectrogram in a heat map, where hotter color represents higher FFT amplitude. To build the heat map for the spectrogram, we use the first 80 FFT magnitude of each chunk to represent the energy in the frequency range of 0∼146 Hz. Spectrogram Enhancement After a spectrogram is generated, we apply spectrogram enhancement techniques to further reduce the noise in the spectrogram. First, we get the energy level for each chunk by summing the magnitude of the first 80 FFT points. We ignore chunks with energy level lower than an empirical threshold so that silent periods are removed. We normalize the FFT magnitudes for each remaining chunk by dividing them with the energy level of the chunk. Second, we apply frequency domain denoising method by subtracting the noise floor from the spectrogram [7]. The noise floor level is estimated through a long-term mean over the spectrograms, e.g., average magnitude over 4 seconds. If the resulting magnitude becomes negative after subtraction, we set it to zero. Third, we superimpose the spectrograms generated by the first twenty PCA results by adding up the magnitude of corresponding time-frequency blocks. Fourth, we apply a 2- dimensional Gaussian low-pass filter, with a size of 5 and σ of 0.8, on the resulting spectrogram. Finally, we obtain a high quality spectrogram that gives detailed information for the walking process of a human subject. FEATURE EXTRACTION In this section, we present how gait features can be extracted from the CSI spectrogram. The gait features include walking speed, gait cycle time, footstep length, movement speeds of torso and legs, and spectrogram signatures
ment is detected when the variance of the PCA result substan- 80 tially deviates from the average noise level.Once the move- ment is detected,we use the torso speed to find the period that 60 the person walks with a steady speed. Estimating Gait Cycle Time We now estimate gait cycle time,which is defined as the time duration between two consecutive events that the right heel Time (seconds) touches the ground.Although we can roughly count steps Figure 3.CSI Spectrogram with human walking. from Figure 3,we need a robust estimation scheme that works Understanding CSI Spectrogram for various walking styles.In WifiU,we use the upper con- tour for the torso reflection to estimate the gait cycle time. We observe that CSI spectrograms give us similar informa- The torso contour frequency fic(t)is defined as: tion as the expensive Doppler radars [16,22,27].Thus,we can extract human gait patterns from CSI spectrograms using fic(t)=max Ff,)一>Y (2) insights from Doppler radars.In particular,we can observe ∑aFf,t) the moving speed changes of different body parts,e.g.,torso and legs,from the spectrogram.Figure 3 shows the spec- where F(f,t)is the magnitude of FFT at time t and frequency trogram obtained in a scenario where a human walks on a f.The summation of m F(f,t)gives the total energy in straight line for a distance of 5.5 meters.Since the torso has the frequency range of 0~fimar.Thus,fic(t)is the highest a larger reflection area than other body parts,the signal en- frequency that has an energy ratio larger than y for the FFT ergy reflected from the torso is the strongest component in the result at time t.We set the threshold y as 5%in our sys- spectrogram.We observe a high energy ("hot"colored)band, tem.After smoothing the resulting fic(t)with a low-pass which corresponds to torso reflection,in the spectrogram in filter,we get the torso contour speed vre(t)as the thick line Figure 3.From the frequency change in the torso reflection in Figure 4(a)using the equation vre(t)=fic(t)/2.Com- in Figure 3,we learn that the subject starts walking at the paring the curve in Figure 4(a)with the spectrogram in Fig- time of 0.5 seconds,speeds up,and then walks with a steady ure 3,we observe that our method reliably tracks the torso speed between the time of 1.5~5 seconds,finally he slows speed variations during the walking periods.The reason for down and stops at the time of 6 seconds.During the walking choosing the contour rather than the frequency that has max- process,the torso reflection has a frequency range of 27~46 imum energy is due to its robustness to noises.With a small Hz,which corresponds to 0.70~1.18 m/s moving speed.We amount of noise,as in the 3~4 seconds period in Figure 3. observe that the torso moving speed fluctuates regularly for the frequency with maximum energy,i.e.,the"hottest"fre- about twice per second.This type of regular fluctuations in quency for a given time,may change significantly and makes torso speed is also observed in Doppler radar results [27] the measurement unreliable and discontinuous in time. The"blue flames"in the frequency region of 40~60 Hz are We use the autocorrelation of the torso contour curve to ro- the reflections of swinging legs [25].The magnitude of leg bustly estimate gait cycle time,which is calculated as: reflections are weak due to the much smaller reflection area R()=>(vic(t)-H)(vic(t-t)-H) (3) of legs.Both the regular speed changes in leg and body reflec- tion indicate that the gait cycle,which contains two footsteps, where u is the average speed of the torso contour.The auto- has a duration of about 1 second.With these basic under- correlation is taken on the period of steady walking,which standings of the spectrogram,we are ready to extract useful is defined as the period where the torso speed(calculated in information about the walking pattern. Section 4.4)is no less than 80%of the maximum torso speed. By taking the autocorrelation,we can find the period of the Detecting Start of Walking curve by comparing the curve to a version of itself that is displaced by t in time.This gives a better estimation than We detect the start of walking based on the fact that mov- directly searching for peaks on the torso contour curve. ing humans generate higher variances in CSI measurements than background noises.When there are no human move- Compared to the original torso contour curve in Figure 4(a),it ments,the variances in CSI measurements are mostly caused is easier to find the peaks in the autocorrelation result in Fig- by noises.Since the noise level changes slowly over time,we ure 4(b).Each peak in autocorrelation means that the contour use a dynamic thresholding algorithm to track the noise level. curve remains similar to the original version when displaced We first calculate the variance var(t)for a segment with 500 by a time of t.As the gait cycle contains two footsteps,the samples at time t in the second PCA component derived in torso speed varies twice in each gait cycle.So,the torso con- Section 3.We do not use the first PCA component because tour curve is similar when displaced by L/2,where L is the it may contain a large amount of noise [31].We update the gait cycle time.In Figure 4(b),the first peak in the autocor- noise level estimation N(t)using an Exponential Moving Av- relation appears at the point=0.57 second;thus,we obtain erage algorithm during the silent period: an estimation for the gait cycle time L=2t as 1.14 seconds. N(t)=(1-an)N(t-1)+an x var(t) (1) Estimating Torso and Leg Speeds where the coefficient an is set to 0.1.We set the detection We estimate the torso and leg speed using the percentile threshold as three times the noise level N(t)so that a move- method developed for Doppler radars [27].The percentile
Figure 3. CSI Spectrogram with human walking. Understanding CSI Spectrogram We observe that CSI spectrograms give us similar information as the expensive Doppler radars [16, 22, 27]. Thus, we can extract human gait patterns from CSI spectrograms using insights from Doppler radars. In particular, we can observe the moving speed changes of different body parts, e.g., torso and legs, from the spectrogram. Figure 3 shows the spectrogram obtained in a scenario where a human walks on a straight line for a distance of 5.5 meters. Since the torso has a larger reflection area than other body parts, the signal energy reflected from the torso is the strongest component in the spectrogram. We observe a high energy (“hot” colored) band, which corresponds to torso reflection, in the spectrogram in Figure 3. From the frequency change in the torso reflection in Figure 3, we learn that the subject starts walking at the time of 0.5 seconds, speeds up, and then walks with a steady speed between the time of 1.5∼5 seconds, finally he slows down and stops at the time of 6 seconds. During the walking process, the torso reflection has a frequency range of 27∼46 Hz, which corresponds to 0.70∼1.18 m/s moving speed. We observe that the torso moving speed fluctuates regularly for about twice per second. This type of regular fluctuations in torso speed is also observed in Doppler radar results [27]. The “blue flames” in the frequency region of 40∼60 Hz are the reflections of swinging legs [25]. The magnitude of leg reflections are weak due to the much smaller reflection area of legs. Both the regular speed changes in leg and body reflection indicate that the gait cycle, which contains two footsteps, has a duration of about 1 second. With these basic understandings of the spectrogram, we are ready to extract useful information about the walking pattern. Detecting Start of Walking We detect the start of walking based on the fact that moving humans generate higher variances in CSI measurements than background noises. When there are no human movements, the variances in CSI measurements are mostly caused by noises. Since the noise level changes slowly over time, we use a dynamic thresholding algorithm to track the noise level. We first calculate the variance var(t) for a segment with 500 samples at time t in the second PCA component derived in Section 3. We do not use the first PCA component because it may contain a large amount of noise [31]. We update the noise level estimation N(t) using an Exponential Moving Average algorithm during the silent period: N(t) = (1−αn)N(t −1) +αn ×var(t) (1) where the coefficient αn is set to 0.1. We set the detection threshold as three times the noise level N(t) so that a movement is detected when the variance of the PCA result substantially deviates from the average noise level. Once the movement is detected, we use the torso speed to find the period that the person walks with a steady speed. Estimating Gait Cycle Time We now estimate gait cycle time, which is defined as the time duration between two consecutive events that the right heel touches the ground. Although we can roughly count steps from Figure 3, we need a robust estimation scheme that works for various walking styles. In WifiU, we use the upper contour for the torso reflection to estimate the gait cycle time. The torso contour frequency ftc(t) is defined as: ftc(t) = max( f F(f,t) ∑ fmax 0 F(f,t) > γ ) (2) where F(f,t) is the magnitude of FFT at time t and frequency f . The summation of ∑ fmax 0 F(f,t) gives the total energy in the frequency range of 0 ∼ fmax. Thus, ftc(t) is the highest frequency that has an energy ratio larger than γ for the FFT result at time t. We set the threshold γ as 5% in our system. After smoothing the resulting ftc(t) with a low-pass filter, we get the torso contour speed vtc(t) as the thick line in Figure 4(a) using the equation vtc(t) = ftc(t)λ/2. Comparing the curve in Figure 4(a) with the spectrogram in Figure 3, we observe that our method reliably tracks the torso speed variations during the walking periods. The reason for choosing the contour rather than the frequency that has maximum energy is due to its robustness to noises. With a small amount of noise, as in the 3∼4 seconds period in Figure 3, the frequency with maximum energy, i.e., the “hottest” frequency for a given time, may change significantly and makes the measurement unreliable and discontinuous in time. We use the autocorrelation of the torso contour curve to robustly estimate gait cycle time, which is calculated as: R(τ) = ∑t (vtc(t)− µ) (vtc(t −τ)− µ) (3) where µ is the average speed of the torso contour. The autocorrelation is taken on the period of steady walking, which is defined as the period where the torso speed (calculated in Section 4.4) is no less than 80% of the maximum torso speed. By taking the autocorrelation, we can find the period of the curve by comparing the curve to a version of itself that is displaced by τ in time. This gives a better estimation than directly searching for peaks on the torso contour curve. Compared to the original torso contour curve in Figure 4(a), it is easier to find the peaks in the autocorrelation result in Figure 4(b). Each peak in autocorrelation means that the contour curve remains similar to the original version when displaced by a time of τ. As the gait cycle contains two footsteps, the torso speed varies twice in each gait cycle. So, the torso contour curve is similar when displaced by L/2, where L is the gait cycle time. In Figure 4(b), the first peak in the autocorrelation appears at the point τ = 0.57 second; thus, we obtain an estimation for the gait cycle time L = 2τ as 1.14 seconds. Estimating Torso and Leg Speeds We estimate the torso and leg speed using the percentile method developed for Doppler radars [27]. The percentile
45 t(seconds) 2 25 Estimation difference(seconds) (a)Speed estimation (b)Autocorrelation of torso contour Figure 5.Estimation difference:CSI vs.accelero- Figure 4.Cycle time and movement speed estimation. meter at a given frequency f is defined as: Sūbjects Height(cm) Cycle (s) Speed (m/s) 165 1.0470.04) 0.972(0.06) F(f,) 160 1.139(0.03) 0.844(0.04) P(f,t)= (4) ∑mFf,d) C 180 1.142(0.05) 0.984(0.06) D 157 0.970(0.03) 1.091(0.04) where P(f,t)is the cumulated percentage of energy for fre- E 177 1.252(0.10) 0.832(0.08) quencies lower than f compared to the total energy of the FFT Table 1.Measurements of testing subjects. result at time t. time of different persons.We choose to use accelerometers The torso movement speed is estimated through the low- to estimate the cycle times instead of estimating them from est frequency values that satisfy P(f,t)>50%,and the leg videos of users for two reasons.First,accelerometers have a speed is estimated by the lowest frequency values that sat- higher time resolution(50 to 100 samples per second)com- isfy P(f,t)>95%.We choose the two threshold values 50% pared to videos(24 to 30 frames per second)[17].Second, and 95%as they were used in prior work on human move- accelerometers attached to ankles provide reliable gait cycle ment modeling using Doppler radar signals [27].The estima- time measurements [9]. tion result is shown in Figure 4(a).The torso speed estimated through the percentile method has smaller variance than the We use the measurements for five subjects to illustrate that torso contour method used in cycle estimation.The smaller our measurement accuracy is good enough for human recog- variance has both advantages and disadvantages.The advant- nition.We asked five human subjects to walk in their natural age is that a smaller variance gives more stable torso speed way for 50 times along a line ofof 5.5 meters.The measure- estimations so that we can reliably detect the walking period ment results are summarized in Table 1.For gait cycle time when the torso speed is stable and falls within the normal and torso speed,the first value in the table is the mean over walking speed range.The disadvantage is that no period- the 50 walks and the value following in the parenthesis is the ical fluctuations can be detected on the torso speed curve ob- standard deviation.We see that people actually walk in differ- tained by the percentile method.Thus,it is suitable to use this ent ways that can be captured by WiFi signals.Their average method for walking speed estimation,but not for cycle time walking speed ranges from 0.832 m/s to 1.091 m/s,while the estimation.This percentile method also allows us to estim- standard deviation of the walking speed for the same person ate leg movement speed.We can see the periodical speed up is very small (less than 0.08 m/s)compared to the differences of legs in Figure 4(a).The leg speeds are useful for human between persons.We also have similar observations in their recognition.But,we do not estimate gait cycle time based gait cycle time distributions.Note that the walking speed on leg speeds because the leg reflections are weak and some- could be slightly lower than the normal speed for users,as times buried in the noise. they only walk for a distance of 5.5 meters in the experiments. Accuracy and Distinguishability More interestingly,even for people with apparently similar The gait cycle time measurements that we obtained from speed,e.g.,human subjects A and C,we still can reliably WiFi signals have comparable accuracy to those obtained separate them using gait cycle time and torso speed.Figure from wearable sensors.We use the accelerometer in a smart- 6 shows the scatter plot for all walking samples from the five phone as a reference to verify the accuracy of the gait cycle subjects.Note that the X-axis is the cadence,defined as the time obtained from spectrograms.We attach the smartphone gait cycles the person takes per second (1/L).We observe on the ankle of the subject so that the accelerometer can get that the five human subjects are clearly separable in Figure 6 a clear measurement on gait cycles.Both the accelerometer and their samples are scattered around the two straight lines. and the CSI measurements are gathered from 100 walking in- The reason for this type of distribution is that people tend to stances of two different subjects.For each walking instance, have constant footstep lengths during walking.For example, we compare the estimated gait cycle time obtained from CSI subject C walks at speeds between 0.88~1.08 meters per with that obtained from the accelerometer.The average dif- second(m/s),which spans a quite large speed range.The ference between the two methods is 11.9 ms.As the spectro- distribution in Figure 6 indicates that the walking speed and gram has a time resolution of 12.8 ms,this difference is very cadence either both are high or both are low.This implies small.The robustness of our measurement method is also sat- that the human subject keeps his footstep length nearly con- isfactory.Figure 5 shows the CDF for the measurement dif- stant,but has small variations in his cadence so that his speed ference.For 80%of the cases,the difference between the two is linearly related to the cadence.Therefore,the few outliers methods is smaller than 50 ms and the maximal difference is for subject E in the upper right corner are possibly not caused less than 90 ms,which is smaller than the differences in cycle by measurement errors.It is more likely that the human sub
0 1 2 3 4 5 6 0 0.5 1 1.5 2 Time (Seconds) Speed (m/s) 50% percentile 95% percentile Torso contour (a) Speed estimation 0.5 1 1.5 2 2.5 3 −400 −200 0 200 400 τ (seconds) Autocorrelation value Autocorrelation of torso contour Peaks for autocorrelation L/2 L/2 L/2 (b) Autocorrelation of torso contour Figure 4. Cycle time and movement speed estimation. 0 0.02 0.04 0.06 0.08 0.1 0 0.2 0.4 0.6 0.8 1 Estimation difference (seconds) CDF Figure 5. Estimation difference: CSI vs. accelerometer at a given frequency f is defined as: P(f,t) = ∑ f 0 F(f,t) ∑ fmax 0 F(f,t) , (4) where P(f,t) is the cumulated percentage of energy for frequencies lower than f compared to the total energy of the FFT result at time t. The torso movement speed is estimated through the lowest frequency values that satisfy P(f,t) ≥ 50%, and the leg speed is estimated by the lowest frequency values that satisfy P(f,t) ≥ 95%. We choose the two threshold values 50% and 95% as they were used in prior work on human movement modeling using Doppler radar signals [27]. The estimation result is shown in Figure 4(a). The torso speed estimated through the percentile method has smaller variance than the torso contour method used in cycle estimation. The smaller variance has both advantages and disadvantages. The advantage is that a smaller variance gives more stable torso speed estimations so that we can reliably detect the walking period when the torso speed is stable and falls within the normal walking speed range. The disadvantage is that no periodical fluctuations can be detected on the torso speed curve obtained by the percentile method. Thus, it is suitable to use this method for walking speed estimation, but not for cycle time estimation. This percentile method also allows us to estimate leg movement speed. We can see the periodical speed up of legs in Figure 4(a). The leg speeds are useful for human recognition. But, we do not estimate gait cycle time based on leg speeds because the leg reflections are weak and sometimes buried in the noise. Accuracy and Distinguishability The gait cycle time measurements that we obtained from WiFi signals have comparable accuracy to those obtained from wearable sensors. We use the accelerometer in a smartphone as a reference to verify the accuracy of the gait cycle time obtained from spectrograms. We attach the smartphone on the ankle of the subject so that the accelerometer can get a clear measurement on gait cycles. Both the accelerometer and the CSI measurements are gathered from 100 walking instances of two different subjects. For each walking instance, we compare the estimated gait cycle time obtained from CSI with that obtained from the accelerometer. The average difference between the two methods is 11.9 ms. As the spectrogram has a time resolution of 12.8 ms, this difference is very small. The robustness of our measurement method is also satisfactory. Figure 5 shows the CDF for the measurement difference. For 80% of the cases, the difference between the two methods is smaller than 50 ms and the maximal difference is less than 90 ms, which is smaller than the differences in cycle Subjects Height (cm) Cycle (s) Speed (m/s) A 163 1.047 (0.04) 0.972 (0.06) B 160 1.139 (0.03) 0.844 (0.04) C 180 1.142 (0.05) 0.984 (0.06) D 157 0.970 (0.03) 1.091 (0.04) E 177 1.252 (0.10) 0.832 (0.08) Table 1. Measurements of testing subjects. time of different persons. We choose to use accelerometers to estimate the cycle times instead of estimating them from videos of users for two reasons. First, accelerometers have a higher time resolution (50 to 100 samples per second) compared to videos (24 to 30 frames per second) [17]. Second, accelerometers attached to ankles provide reliable gait cycle time measurements [9]. We use the measurements for five subjects to illustrate that our measurement accuracy is good enough for human recognition. We asked five human subjects to walk in their natural way for 50 times along a line of of 5.5 meters. The measurement results are summarized in Table 1. For gait cycle time and torso speed, the first value in the table is the mean over the 50 walks and the value following in the parenthesis is the standard deviation. We see that people actually walk in different ways that can be captured by WiFi signals. Their average walking speed ranges from 0.832 m/s to 1.091 m/s, while the standard deviation of the walking speed for the same person is very small (less than 0.08 m/s) compared to the differences between persons. We also have similar observations in their gait cycle time distributions. Note that the walking speed could be slightly lower than the normal speed for users, as they only walk for a distance of 5.5 meters in the experiments. More interestingly, even for people with apparently similar speed, e.g., human subjects A and C, we still can reliably separate them using gait cycle time and torso speed. Figure 6 shows the scatter plot for all walking samples from the five subjects. Note that the X-axis is the cadence, defined as the gait cycles the person takes per second (1/L). We observe that the five human subjects are clearly separable in Figure 6 and their samples are scattered around the two straight lines. The reason for this type of distribution is that people tend to have constant footstep lengths during walking. For example, subject C walks at speeds between 0.88 ∼ 1.08 meters per second (m/s), which spans a quite large speed range. The distribution in Figure 6 indicates that the walking speed and cadence either both are high or both are low. This implies that the human subject keeps his footstep length nearly constant, but has small variations in his cadence so that his speed is linearly related to the cadence. Therefore, the few outliers for subject E in the upper right corner are possibly not caused by measurement errors. It is more likely that the human sub-
ture the detailed walking patterns of a human subject.In other words,the energy distribution can serve as the"signature"for a human gait pattern. We extract the spectrogram signature as follows.First,we use the cycle time measured in the previous section to help us to find peaks on the torso contour curve.We then cut the spec- trograms into half-gait-cycles using these peaks.Each half- ce (cycles per gait-cycle contains the process of swinging one leg,which cond) could be the left or the right leg.We further divide the half- Figure 6.Distribution of cadence and torso speed. gait-cycle into 4 stages with equal length in time,which rep- resent the different stages for leg swinging.On each stage, we calculate the normalized energy on 40 frequency points by 人A0A人 averaging the FFT magnitudes during the stage.The shape of energy change in the frequency domain serves as the spectro- gram signature for the subject. The spectrogram signature can serve as a fingerprint of the Figure 7.Spectrogram signatures for five human subjects on stage 2 of gait pattern.Figure 7 plots signatures on stage 2 for five sub- the half-gait-cvcle. jects.For each subject,we plot the signature curve for 50 walking samples on the same graph.We observe that the sig- ject changes his walking speed for some tests,as the walking nature curves for the same subject are clustered together so speed actually increase linearly with the cadence for these that they appear to be a"thick"curve,while different subjects outliers.As tall people tend to have longer step lengths,we have very different signatures.These signatures are determ- also give the heights of these five subjects in Table 1.The ined by the gait patterns and other complex factors such as samples for subject C and E appear on a line with different the height and size of the person.Therefore,they can serve slope in Figure 6 than those for subject A,B and D because as unique characters for a human they are taller and have longer step lengths than others In WifU,we extract a set of 170 features for each walking To understand the relationship between footstep length and sample,including gait cycle length,estimated footstep length, height,we further run the footstep length estimation on a lar- the maximum,minimum,average,and variance for torso and ger data set that contains 49 persons,using the product of leg speeds during the gait cycle,and spectrogram signatures torso speed and cycle time as the estimator of the step size. for the 4 stages in the half-gait-cycle on 40 frequency points. The Pearson correlation coefficient between the height and the footstep length of the subject is 0.405 and the p-value is TRAINING AND CLASSIFICATION 0.0039.This shows that there is actually a positive correla- In this section,based on the features that we extract from tion between the estimated footstep length and the height of WiFi signals,we use machine learning techniques to build the subject.Note that due to the variance in human behavior, a gait recognition system.The first step for human recog- there are subjects that varies footstep length significantly dur- nition is enrollment.In this step,we collect gait instances ing the test and some subject have non proportional step size of the target human subject,which will be used as training to their heights.Nevertheless,the footstep length still can data.Our experimental results show that the number of gait serve as a useful feature for gait based recognition instances that WifiU needs is around 40.Although the target human subject needs to walk for a distance of 5~6 meters for Spectrogram Signatures 40 times in the enrollment phase,we can collect training data We propose a new set of features,called spectrogram signa- when the individual is using traditional identification mech- tures,to further describe the gait patterns in detail.As hu- anisms(such as access tokens)to identify himself to reduce mans may walk with similar speed and cadence,simply using the data collection effort. the two metrics of gait cycle time and torso speed may not sufficiently recognize an individual among a large candidate Using the data gathered in enrollment phase,WifiU trains a set.The spectrogram in Figure 3 actually provides more in- gait model for the target human subject,which can be either used for single user identification or multiple user recogni- formation than the gait cycle time and torso speed.For ex- tion applications.The single user identification application ample,the small green downward spikes appeared in almost every cycle,e.g.,at 1.7 and 2.3 seconds,are possibly caused answers the question whether the subject is the target person or a stranger and the multiple user recognition application an- by the lower leg or the foot of the supporting leg that has swers the question who the subject is among the given set of lower speed than the torso [27].Such detailed information enables us to characterize human gait patterns.To capture users.We use the LibSVM tool [5]with the Radial Basis human gait patterns from spectrograms,we use the distribu- Function(RBF)kernel in the training.The optimal values for parameters v and y for the RBF kernel are selected through tion of reflected energy on predetermined frequency points to serve as"signatures"of the spectrogram.These energy dis- grid search. tributions give an overview on how different body parts are To train a gait model for single user identification,we build a moving at a given stage of walking.Thus,they help to cap- classifier that can classify gait instances into two classes,one
0.7 0.8 0.9 1 1.1 0.7 0.8 0.9 1 1.1 Cadence (cycles per second) Torso speed (m/s) A B C D E Figure 6. Distribution of cadence and torso speed. 50 100 0 0.05 0.1 Freq (Hz) Energy Subject A 50 100 0 0.05 0.1 Freq (Hz) Energy Subject B 50 100 0 0.05 0.1 Freq (Hz) Energy Subject C 50 100 0 0.05 0.1 Freq (Hz) Energy Subject D 50 100 0 0.05 0.1 Freq (Hz) Energy Subject E Figure 7. Spectrogram signatures for five human subjects on stage 2 of the half-gait-cycle. ject changes his walking speed for some tests, as the walking speed actually increase linearly with the cadence for these outliers. As tall people tend to have longer step lengths, we also give the heights of these five subjects in Table 1. The samples for subject C and E appear on a line with different slope in Figure 6 than those for subject A, B and D because they are taller and have longer step lengths than others. To understand the relationship between footstep length and height, we further run the footstep length estimation on a larger data set that contains 49 persons, using the product of torso speed and cycle time as the estimator of the step size. The Pearson correlation coefficient between the height and the footstep length of the subject is 0.405 and the p-value is 0.0039. This shows that there is actually a positive correlation between the estimated footstep length and the height of the subject. Note that due to the variance in human behavior, there are subjects that varies footstep length significantly during the test and some subject have non proportional step size to their heights. Nevertheless, the footstep length still can serve as a useful feature for gait based recognition. Spectrogram Signatures We propose a new set of features, called spectrogram signatures, to further describe the gait patterns in detail. As humans may walk with similar speed and cadence, simply using the two metrics of gait cycle time and torso speed may not sufficiently recognize an individual among a large candidate set. The spectrogram in Figure 3 actually provides more information than the gait cycle time and torso speed. For example, the small green downward spikes appeared in almost every cycle, e.g., at 1.7 and 2.3 seconds, are possibly caused by the lower leg or the foot of the supporting leg that has lower speed than the torso [27]. Such detailed information enables us to characterize human gait patterns. To capture human gait patterns from spectrograms, we use the distribution of reflected energy on predetermined frequency points to serve as “signatures” of the spectrogram. These energy distributions give an overview on how different body parts are moving at a given stage of walking. Thus, they help to capture the detailed walking patterns of a human subject. In other words, the energy distribution can serve as the “signature” for a human gait pattern. We extract the spectrogram signature as follows. First, we use the cycle time measured in the previous section to help us to find peaks on the torso contour curve. We then cut the spectrograms into half-gait-cycles using these peaks. Each halfgait-cycle contains the process of swinging one leg, which could be the left or the right leg. We further divide the halfgait-cycle into 4 stages with equal length in time, which represent the different stages for leg swinging. On each stage, we calculate the normalized energy on 40 frequency points by averaging the FFT magnitudes during the stage. The shape of energy change in the frequency domain serves as the spectrogram signature for the subject. The spectrogram signature can serve as a fingerprint of the gait pattern. Figure 7 plots signatures on stage 2 for five subjects. For each subject, we plot the signature curve for 50 walking samples on the same graph. We observe that the signature curves for the same subject are clustered together so that they appear to be a “thick” curve, while different subjects have very different signatures. These signatures are determined by the gait patterns and other complex factors such as the height and size of the person. Therefore, they can serve as unique characters for a human. In WifiU, we extract a set of 170 features for each walking sample, including gait cycle length, estimated footstep length, the maximum, minimum, average, and variance for torso and leg speeds during the gait cycle, and spectrogram signatures for the 4 stages in the half-gait-cycle on 40 frequency points. TRAINING AND CLASSIFICATION In this section, based on the features that we extract from WiFi signals, we use machine learning techniques to build a gait recognition system. The first step for human recognition is enrollment. In this step, we collect gait instances of the target human subject, which will be used as training data. Our experimental results show that the number of gait instances that WifiU needs is around 40. Although the target human subject needs to walk for a distance of 5∼6 meters for 40 times in the enrollment phase, we can collect training data when the individual is using traditional identification mechanisms (such as access tokens) to identify himself to reduce the data collection effort. Using the data gathered in enrollment phase, WifiU trains a gait model for the target human subject, which can be either used for single user identification or multiple user recognition applications. The single user identification application answers the question whether the subject is the target person or a stranger and the multiple user recognition application answers the question who the subject is among the given set of users. We use the LibSVM tool [5] with the Radial Basis Function (RBF) kernel in the training. The optimal values for parameters ν and γ for the RBF kernel are selected through grid search. To train a gait model for single user identification, we build a classifier that can classify gait instances into two classes, one
for the target person and one for others.In the training pro- cess,we use the gait instances from 7~9 benchmark persons as training data for the negative class.In real deployment,gait instances of benchmark persons can be drawn from a stand- ard database.The reason for using benchmark persons is that these instances are helpful to determine the decision bound- Distance(meters) ary for the target person and improve the identification accur- Figure 8.Detection ratio vs.operational distances acy.Once the gait model is trained,the classifier can calculate collected 50-60 walking instances for each human subject. the fitness probability that an unknown gait instance belongs With the consent of all human subjects,we also recorded their to the target person.We treat gait instances with fitness prob- heights and genders in our anonymized database. ability higher than a given threshold as instances belonging to the target person.The classifier can identify gait instances Evaluation Metrics for persons that are not seen in the training phase,as their gait We evaluated WifiU from four perspectives:operational dis- features have low fitness probability to the gait model of the tance,effectiveness,robustness,and efficiency.For opera- target person tional distance,we evaluated the minimum distance between To recognize a person in a given set of candidates,we need the walking human subject and the WiFi devices.For accur- one gait model for each person in the candidate set.For acy,we evaluated identification accuracy for single user iden- each person,we train a two-class classifier that can separ- tification and top-k accuracy for multiple user recognition. ate him/her from all the other candidates in the candidate set. We evaluated the identification accuracy in terms of False Ac- Thus,for a candidate set with M persons,we build M one- ceptance Rate (FAR)and False Rejection Rate (FRR).The vs-all classifiers.In the prediction phase,we fit the unknown FAR is defined as the rate that a stranger is wrongly clas- walking sample into the M classifiers and get the probability sified as the target subject and the FRR is the rate that the of fitness.WifiU selects the models with the k highest fitness true target is wrongly classified as a stranger.Since we can probability as the top-k candidates for the testing sample. tradeoff between the FAR and FRR by changing the probabil- ity threshold for identification,we define the Equal Error Rate EXPERIMENTAL RESULTS (EER)point as the point that FAR and FRR are equal.Top-k Data Sets accuracy is defined as the percentage of tested instances in We collected gait patterns from 50 human subject with IRB which WifiU is correct in declaring that the walking human approval.The human subjects were 36 male and 14 female subject is among the top-k candidates.Note that WifiU re- graduate students,with similar ages in the range of 22-25 ports a ranked list of candidates in the decreasing order of years.We conducted our experiments in a typical lab with similarity with the walking human subject.For robustness, an area of 50 square meters.The layout of this lab is plot- we evaluated the identification accuracy of WifiU from two ted in Figure 1(b).The WiFi sender and the WiFi receiver perspectives:(1)effect of evolution of human gait with time were placed on a table with a height of 80 cm and they were and (2)effect of difference in apparel and accessories.For separated by a distance of 1.6 meters.In our experiments, efficiency,we focused on evaluating classification efficiency, we used a NetGear JR6100 WiFi router (of less than 100 which is the time that the classifier takes to make the classi- USD)that supports IEEE 802.11n protocol as the sender and fication decision,and classifier construction efficiency,which a Thinkpad X200 laptop with the Intel 5300 wireless card (of is the time for constructing the classifier. about 10 USD)as the receiver to collect CSI measurements using the Linux CSI tool [11].The wireless router was con- Operational Distance figured to work at 5 GHz band and used a channel bandwidth Our results show that WifiU can detect a walking human sub- of 20 MHz.We chose the 5 GHz band.rather than 2.4 GHz ject at a range as long as 14 meters.Using the movement de- band because the wavelength of 5 GHz is shorter,and shorter tection algorithm described in Section 4.2.we measured the wavelengths give better resolutions in movement speeds.The detection range of our system in a large open lobby area.Fig- settings of the WiFi router,including channel bandwidth,car- ure 8 shows the detection probability for a walking human at rier frequency and data rate settings,were similar to those different distances.We observe that our system is able to de- of our campus network where other WiFi devices coexisted tect a walking human with an accuracy of 92%at a distance of in the same channel.We used the omni-directional antennas 14 m and the accuracy quickly reduces to around 50%at the that come with the router and laptop without any modifica- distance of 16 m.However,the range that our system can re- tions.We used the default transmission power settings of the liably extract gait information is smaller.In our experiments WiFi devices,so there are no potential harms to human sub- we found the operational distance that allows us to perform jects,as our devices fully comply with FCC regulations.We gait recognition is about 6 meters,which is the distance used anonymized collected data to protect the privacy of human in our data collection process. subjects Accuracy The detailed data collection process is as follows.Each sub- To evaluate identification accuracy,we first randomly choose ject was requested to walk repeatedly on a straight line with 7 subjects from the 50 subjects in the database to serve as the distance of 5.5 m.They were asked to walk in their natural benchmark set.For each target subject not in the benchmark way without intentional speed up or slow down.The CSI val- set,we train a SVM classifier that treats the target subject ues were recorded on the laptop and processed offline.We as a class and benchmark subjects as the other class.The
for the target person and one for others. In the training process, we use the gait instances from 7∼9 benchmark persons as training data for the negative class. In real deployment, gait instances of benchmark persons can be drawn from a standard database. The reason for using benchmark persons is that these instances are helpful to determine the decision boundary for the target person and improve the identification accuracy. Once the gait model is trained, the classifier can calculate the fitness probability that an unknown gait instance belongs to the target person. We treat gait instances with fitness probability higher than a given threshold as instances belonging to the target person. The classifier can identify gait instances for persons that are not seen in the training phase, as their gait features have low fitness probability to the gait model of the target person. To recognize a person in a given set of candidates, we need one gait model for each person in the candidate set. For each person, we train a two-class classifier that can separate him/her from all the other candidates in the candidate set. Thus, for a candidate set with M persons, we build M onevs-all classifiers. In the prediction phase, we fit the unknown walking sample into the M classifiers and get the probability of fitness. WifiU selects the models with the k highest fitness probability as the top-k candidates for the testing sample. EXPERIMENTAL RESULTS Data Sets We collected gait patterns from 50 human subject with IRB approval. The human subjects were 36 male and 14 female graduate students, with similar ages in the range of 22-25 years. We conducted our experiments in a typical lab with an area of 50 square meters. The layout of this lab is plotted in Figure 1(b). The WiFi sender and the WiFi receiver were placed on a table with a height of 80 cm and they were separated by a distance of 1.6 meters. In our experiments, we used a NetGear JR6100 WiFi router (of less than 100 USD) that supports IEEE 802.11n protocol as the sender and a Thinkpad X200 laptop with the Intel 5300 wireless card (of about 10 USD) as the receiver to collect CSI measurements using the Linux CSI tool [11]. The wireless router was con- figured to work at 5 GHz band and used a channel bandwidth of 20 MHz. We chose the 5 GHz band, rather than 2.4 GHz band because the wavelength of 5 GHz is shorter, and shorter wavelengths give better resolutions in movement speeds. The settings of the WiFi router, including channel bandwidth, carrier frequency and data rate settings, were similar to those of our campus network where other WiFi devices coexisted in the same channel. We used the omni-directional antennas that come with the router and laptop without any modifications. We used the default transmission power settings of the WiFi devices, so there are no potential harms to human subjects, as our devices fully comply with FCC regulations. We anonymized collected data to protect the privacy of human subjects. The detailed data collection process is as follows. Each subject was requested to walk repeatedly on a straight line with distance of 5.5 m. They were asked to walk in their natural way without intentional speed up or slow down. The CSI values were recorded on the laptop and processed offline. We 2 4 6 8 10 12 14 16 0 0.2 0.4 0.6 0.8 1 Distance (meters) Detection ratio Figure 8. Detection ratio vs. operational distances collected 50–60 walking instances for each human subject. With the consent of all human subjects, we also recorded their heights and genders in our anonymized database. Evaluation Metrics We evaluated WifiU from four perspectives: operational distance, effectiveness, robustness, and efficiency. For operational distance, we evaluated the minimum distance between the walking human subject and the WiFi devices. For accuracy, we evaluated identification accuracy for single user identification and top-k accuracy for multiple user recognition. We evaluated the identification accuracy in terms of False Acceptance Rate (FAR) and False Rejection Rate (FRR). The FAR is defined as the rate that a stranger is wrongly classified as the target subject and the FRR is the rate that the true target is wrongly classified as a stranger. Since we can tradeoff between the FAR and FRR by changing the probability threshold for identification, we define the Equal Error Rate (EER) point as the point that FAR and FRR are equal. Top-k accuracy is defined as the percentage of tested instances in which WifiU is correct in declaring that the walking human subject is among the top-k candidates. Note that WifiU reports a ranked list of candidates in the decreasing order of similarity with the walking human subject. For robustness, we evaluated the identification accuracy of WifiU from two perspectives: (1) effect of evolution of human gait with time and (2) effect of difference in apparel and accessories. For efficiency, we focused on evaluating classification efficiency, which is the time that the classifier takes to make the classi- fication decision, and classifier construction efficiency, which is the time for constructing the classifier. Operational Distance Our results show that WifiU can detect a walking human subject at a range as long as 14 meters. Using the movement detection algorithm described in Section 4.2, we measured the detection range of our system in a large open lobby area. Figure 8 shows the detection probability for a walking human at different distances. We observe that our system is able to detect a walking human with an accuracy of 92% at a distance of 14 m and the accuracy quickly reduces to around 50% at the distance of 16 m. However, the range that our system can reliably extract gait information is smaller. In our experiments we found the operational distance that allows us to perform gait recognition is about 6 meters, which is the distance used in our data collection process. Accuracy To evaluate identification accuracy, we first randomly choose 7 subjects from the 50 subjects in the database to serve as the benchmark set. For each target subject not in the benchmark set, we train a SVM classifier that treats the target subject as a class and benchmark subjects as the other class. The
Our results show that using the benchmark set size of 7 and 08 PCA components size of 20 in WifiU are effective.Figure 0.6 10(b)gives the performance of WifiU under different bench- 8 0.4 mark set sizes.We observe that using more benchmark sub- -FAR jects gives lower EER,but the results for having 7,9,and 02 …FRR 11 benchmark subjects are almost the same.Therefore,we 0.05 吧mor ra 0.2 choose to use 7~9 benchmark subjects in WifiU.Figure 10(c) gives the performance of WifiU under different number of Figure 9.CDF of FAR and FRR PCA components used in spectrogram generation.Using 20 PCA components is enough,since adding more components do not further improve the performance of WifiU. rest 50-7-1 =42 subjects are treated as strangers in the evaluation.Note that none of the walking instances of these Robustness strangers are seen by the SVM during training.The FAR and Our results show that evolution of human gait does not sig- FRR are calculated using 10 fold cross validation,where one nificantly affect the accuracy of WifiU.To study the robust- tenth of the walking instances are used for testing and the ness of WifU against gait evolution,we collected more than remaining are used for training.After getting the individual 200 walking samples from the same subject over a period FAR and FRR for all subjects not in the benchmark set,we of four months and performed 10 fold cross validation on repeat the whole process using a new set of benchmark sub- these samples.Figure 11(a)plots FAR vs.FRR of WifiU jects.The final results are averaged over 100 randomly selec- for this scenario obtained from the 10 fold cross validation ted benchmark sets. We observe from this figure that WifiU achieves an EER of Our results show that WifiU achieves average FAR and FRR just 11.3%on these testing samples.WifiU is also robust to of 8.05%and 9.54%,respectively,when setting the probab- small changes in environments,such as moving chairs/tables ility threshold for identification to 0.5.In other words,our around the room or slightly changing the location of the system determines that the gait sample belongs to the true Router/Laptop,which actually happened frequently during user when the fitness probability is higher than the threshold the four month period of data collection. of 0.5.Figure 9 shows the distribution of FAR and FRR Our results show that the accuracy of WifiU deteriorates only on different subjects.For more than 80%subjects,WifiU when the apparel and accessories of the target subject change achieves FAR smaller than 10%and FRR smaller than 15%. significantly.To study the impact of change in apparel and By changing the probability threshold in the gait model,we accessories on the accuracy of WifiU,we first train WifiU us- can achieve different tradeoffs between FAR and FRR.Fig- ing training samples of a subject wearing winter clothes and ure 10(a)shows the tradeoff curve for FAR and FRR.At the wearing spring clothes,and then evaluate its FRR on testing point where FAR and FRR is equal,we get an EER of 8.6% samples obtained when the same subject was wearing differ- when we use all the available training data. ent types of clothes i.e.,suits or carrying a briefcase.Figure To evaluate top-k accuracy,in each experiment,we randomly 11(b)plots the FRR for different apparel and accessories op- tions when FAR is set at 11.1%.The FRR increases from pickup ten human subjects from our walking database.The top-k accuracy for the given candidate set is obtained via 10 10%in the training set(winter/spring clothes)to at least 25 fold cross validation over the samples of the ten subjects.We when the subject is wearing different apparel and accessor- perform this experiment for 100 times and calculate the av- ies.This happens due to the well studied fact that human erage top-k accuracies.WifiU achieves top-1,top-2,and top- gait changes when wearing different types of apparel and ac- 3 accuracies of 92.31%,97.58%,and 98.86%,respectively. cessories [8].Despite the fact that we never trained WifU on Note that the recognition task is considerably harder than samples with these apparel and accessories,it still provides the identification task in the previous experiment because in decent FRR while keeping FAR as low as 11.1%. the recognition task,WifiU does not just have to determine whether the test subject is the true user or not,rather has to Efficiency recognize exactly which user it is.When we increase the size Our results show that WifiU can run in realtime on laptops of subject set to 50 subjects,the top-1,top-2,and top-3 ac- and desktop computers.We evaluated the efficiency of WifU curacies are 79.28%,89.52%,and 93.05%,respectively. on a laptop equipped with an Intel Core i5 CPU running at 2.8 GHz.Table 2 gives the CPU processing time and the standard Our results show that 40 training instances are practically deviation for processing one second of CSI measurement in large enough to build the classifier as shown by Figure 10(a). our implementation on Matlab.We observe that the total CPU We plot the FAR and FRR for training schemes that use time required to process one second of data is 0.275 second all available instances for training and compare them to the so that the laptop can easily handle realtime processing for schemes that limit the training set size to 20.30 and 40 in- WifiU.The major computational cost (about 71.3%)is in the stances.We see a diminishing improvement in EER with in- PCA denoising step because PCA needs to process the raw creasing number of training instances.Furthermore,the train- CSI measurements that have 180 x 2500 =450k samples per ing set size of 40 gives nearly the same identification accuracy second.As our PCA process reduces the data size to one as the scheme that uses all available training instances,which ninth of the raw CSI measurements,the spectrogram gener- is normally more than 50 instances in our database. ation step only takes 26.2%of the total computational time
0 0.05 0.1 0.15 0.2 0.25 0 0.2 0.4 0.6 0.8 1 Error rate CDF FAR FRR Figure 9. CDF of FAR and FRR rest 50 − 7 − 1 = 42 subjects are treated as strangers in the evaluation. Note that none of the walking instances of these strangers are seen by the SVM during training. The FAR and FRR are calculated using 10 fold cross validation, where one tenth of the walking instances are used for testing and the remaining are used for training. After getting the individual FAR and FRR for all subjects not in the benchmark set, we repeat the whole process using a new set of benchmark subjects. The final results are averaged over 100 randomly selected benchmark sets. Our results show that WifiU achieves average FAR and FRR of 8.05% and 9.54%, respectively, when setting the probability threshold for identification to 0.5. In other words, our system determines that the gait sample belongs to the true user when the fitness probability is higher than the threshold of 0.5. Figure 9 shows the distribution of FAR and FRR on different subjects. For more than 80% subjects, WifiU achieves FAR smaller than 10% and FRR smaller than 15%. By changing the probability threshold in the gait model, we can achieve different tradeoffs between FAR and FRR. Figure 10(a) shows the tradeoff curve for FAR and FRR. At the point where FAR and FRR is equal, we get an EER of 8.6% when we use all the available training data. To evaluate top-k accuracy, in each experiment, we randomly pickup ten human subjects from our walking database. The top-k accuracy for the given candidate set is obtained via 10 fold cross validation over the samples of the ten subjects. We perform this experiment for 100 times and calculate the average top-k accuracies. WifiU achieves top-1, top-2, and top- 3 accuracies of 92.31%, 97.58%, and 98.86%, respectively. Note that the recognition task is considerably harder than the identification task in the previous experiment because in the recognition task, WifiU does not just have to determine whether the test subject is the true user or not, rather has to recognize exactly which user it is. When we increase the size of subject set to 50 subjects, the top-1, top-2, and top-3 accuracies are 79.28%, 89.52%, and 93.05%, respectively. Our results show that 40 training instances are practically large enough to build the classifier as shown by Figure 10(a). We plot the FAR and FRR for training schemes that use all available instances for training and compare them to the schemes that limit the training set size to 20, 30 and 40 instances. We see a diminishing improvement in EER with increasing number of training instances. Furthermore, the training set size of 40 gives nearly the same identification accuracy as the scheme that uses all available training instances, which is normally more than 50 instances in our database. Our results show that using the benchmark set size of 7 and PCA components size of 20 in WifiU are effective. Figure 10(b) gives the performance of WifiU under different benchmark set sizes. We observe that using more benchmark subjects gives lower EER, but the results for having 7, 9, and 11 benchmark subjects are almost the same. Therefore, we choose to use 7∼9 benchmark subjects in WifiU. Figure 10(c) gives the performance of WifiU under different number of PCA components used in spectrogram generation. Using 20 PCA components is enough, since adding more components do not further improve the performance of WifiU. Robustness Our results show that evolution of human gait does not significantly affect the accuracy of WifiU. To study the robustness of WifiU against gait evolution, we collected more than 200 walking samples from the same subject over a period of four months and performed 10 fold cross validation on these samples. Figure 11(a) plots FAR vs. FRR of WifiU for this scenario obtained from the 10 fold cross validation. We observe from this figure that WifiU achieves an EER of just 11.3% on these testing samples. WifiU is also robust to small changes in environments, such as moving chairs/tables around the room or slightly changing the location of the Router/Laptop, which actually happened frequently during the four month period of data collection. Our results show that the accuracy of WifiU deteriorates only when the apparel and accessories of the target subject change significantly. To study the impact of change in apparel and accessories on the accuracy of WifiU, we first train WifiU using training samples of a subject wearing winter clothes and wearing spring clothes, and then evaluate its FRR on testing samples obtained when the same subject was wearing different types of clothes i.e., suits or carrying a briefcase. Figure 11(b) plots the FRR for different apparel and accessories options when FAR is set at 11.1%. The FRR increases from 10% in the training set (winter/spring clothes) to at least 25% when the subject is wearing different apparel and accessories. This happens due to the well studied fact that human gait changes when wearing different types of apparel and accessories [8]. Despite the fact that we never trained WifiU on samples with these apparel and accessories, it still provides decent FRR while keeping FAR as low as 11.1%. Efficiency Our results show that WifiU can run in realtime on laptops and desktop computers. We evaluated the efficiency of WifiU on a laptop equipped with an Intel Core i5 CPU running at 2.8 GHz. Table 2 gives the CPU processing time and the standard deviation for processing one second of CSI measurement in our implementation on Matlab. We observe that the total CPU time required to process one second of data is 0.275 second so that the laptop can easily handle realtime processing for WifiU. The major computational cost (about 71.3%) is in the PCA denoising step because PCA needs to process the raw CSI measurements that have 180×2500 = 450k samples per second. As our PCA process reduces the data size to one ninth of the raw CSI measurements, the spectrogram generation step only takes 26.2% of the total computational time
20u元 O PCA S PC 30 PCA EER I .0 02 0.05 False Reiection Rate 02 0.2分 (a)Different size of training instances (b)Different size of benchmark sets (c)Different size of PCA components Figure 10.Tradeoff between FAR and FRR ase ReconRate Number of Training insnces mber of2 sining ingance (a)FAR and FRR (b)FRR at FAR=11.1% (a)Training time (b)Classification time Figure 11.Performance evaluation with gait evolution Figure 12.Efficiency of training and classification CPU time (s) Time Std(s)Percentage ing speed.Therefore,the current implementation of WifiU PCA 0.196 0038 713 is only suitable for confined spaces,such as a corridor or Spectrogram 0.072 0.018 26.2% Feature 0.007 0.001 2.5% a narrow entrance.Identifying gaits with no restrictions on Classification 1.7×10-5 4.3×10-6 0.006% the walking path would be an interesting problem for future Total 0.275 0.016 100% research.Second,when there are multiple users walking at Table 2.CPU time to process one second CSI data the same time,the gait patterns captured by WifiU are com- plex mixtures of multiple activities of the users.Although The classification step takes a negligible amount of CPU time static users (such as those that were sitting in the same room because SVM classification with two classes is fast. while we collected data)have no impact to the performance of Our results show that WifiU can train a gait model within 100 WifiU,it is difficult to separate the gait signals from multiple milliseconds when the number of training instances is smal- moving users.In future,we plan to explore the possibility of ler than 800.Figure 12(a)and 12(b)shows that the training using multiple devices to separate gait signals from multiple and classification time of WifiU.increases as the number of users. training instances increases,respectively.The training time CONCLUSIONS increases slowly when the number of training instances in- creases.Curve fitting results show that the training time in- We make the following four key contributions.First,we creases at a speed of O(N4),where N is the number of train- demonstrate the feasibility of using WiFi signals from COTS ing instances.The highly efficient training process enables devices for gait recognition.Second,we propose signal pro- WifiU to continuously retrain the gait model when new meas- cessing techniques to convert raw noisy WiFi signals to high- urements are available.The classification time of WifiU only fidelity spectrograms.Third,we propose methods to extract changes slightly from 0.01 ms to 0.02 ms when the number human gait information and individual specific features from of training instances increases from 100 to 800. spectrograms.Last,we build the WifU system that can re- cognize humans based on the extracted features. LIMITATIONS Acknowledgments WifiU establishes the feasibility of using COTS WiFi devices This work is partially supported by the National Natural Sci- for recognizing humans through their gait patterns.How- ence Foundation of China under Grant Numbers 61373129, ever,our current implementation of WifiU has two limita- 61472184.61321491.and 61472185.the National Science tions.First,the users must walk on a predefined path in Foundation under Grant Numbers CNS-1421407,Collabor- a predefined walking direction,e.g.,walking for 5.5 meters ative Innovation Center of Novel Software Technology and on a straight line as in our experiments.The classification Industrialization,and the Jiangsu High-level Innovation and models trained for a given walking path and direction can- Entrepreneurship(Shuangchuang)Program. not be used for testing samples obtained on different walk- ing paths and directions.This is because Doppler spectro- REFERENCES grams are sensitive to the walking directions.When the user 1.Fadel Adib,Chen-Yu Hsu,Hongzi Mao,Dina Katabi, walks on a different path and/or in a different direction with and Fredo Durand.2015a.Capturing the human figure respect to the WiFi transmitter/receiver,the perceived Dop- through a wall.ACM Transactions on Graphics 34,6 pler frequency changes even if the user retains the same walk- (2015),219
0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 False Rejection Rate False Acceptance Rate 20 training instances 30 training instances 40 training instances Full training set EER line (a) Different size of training instances 0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 False Rejection Rate False Acceptance Rate 3 benchmark 5 benchmark 7 benchmark 9 benchmark 11 benchmark EER line (b) Different size of benchmark sets 0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 False Rejection Rate False Acceptance Rate 5 PCA components 10 PCA components 15 PCA components 20 PCA components 25 PCA components 30 PCA components EER line (c) Different size of PCA components Figure 10. Tradeoff between FAR and FRR 0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 False Rejection Rate False Acceptance Rate EER Line (a) FAR and FRR Spring Winter Suits Briefcase 0 0.1 0.2 0.3 0.4 False Rejection Rate (b) FRR at FAR=11.1% Figure 11. Performance evaluation with gait evolution CPU time (s) Time Std (s) Percentage PCA 0.196 0.038 71.3% Spectrogram 0.072 0.018 26.2% Feature 0.007 0.001 2.5% Classification 1.7×10−5 4.3×10−6 0.006% Total 0.275 0.016 100% Table 2. CPU time to process one second CSI data The classification step takes a negligible amount of CPU time because SVM classification with two classes is fast. Our results show that WifiU can train a gait model within 100 milliseconds when the number of training instances is smaller than 800. Figure 12(a) and 12(b) shows that the training and classification time of WifiU, increases as the number of training instances increases, respectively. The training time increases slowly when the number of training instances increases. Curve fitting results show that the training time increases at a speed of O(N 1.4 ), where N is the number of training instances. The highly efficient training process enables WifiU to continuously retrain the gait model when new measurements are available. The classification time of WifiU only changes slightly from 0.01 ms to 0.02 ms when the number of training instances increases from 100 to 800. LIMITATIONS WifiU establishes the feasibility of using COTS WiFi devices for recognizing humans through their gait patterns. However, our current implementation of WifiU has two limitations. First, the users must walk on a predefined path in a predefined walking direction, e.g., walking for 5.5 meters on a straight line as in our experiments. The classification models trained for a given walking path and direction cannot be used for testing samples obtained on different walking paths and directions. This is because Doppler spectrograms are sensitive to the walking directions. When the user walks on a different path and/or in a different direction with respect to the WiFi transmitter/receiver, the perceived Doppler frequency changes even if the user retains the same walk- 200 400 600 800 0 20 40 60 80 100 Number of Training instances Training time (ms) (a) Training time 200 400 600 800 0 0.005 0.01 0.015 0.02 Number of Training instances Classification time (ms) (b) Classification time Figure 12. Efficiency of training and classification ing speed. Therefore, the current implementation of WifiU is only suitable for confined spaces, such as a corridor or a narrow entrance. Identifying gaits with no restrictions on the walking path would be an interesting problem for future research. Second, when there are multiple users walking at the same time, the gait patterns captured by WifiU are complex mixtures of multiple activities of the users. Although static users (such as those that were sitting in the same room while we collected data) have no impact to the performance of WifiU, it is difficult to separate the gait signals from multiple moving users. In future, we plan to explore the possibility of using multiple devices to separate gait signals from multiple users. CONCLUSIONS We make the following four key contributions. First, we demonstrate the feasibility of using WiFi signals from COTS devices for gait recognition. Second, we propose signal processing techniques to convert raw noisy WiFi signals to high- fidelity spectrograms. Third, we propose methods to extract human gait information and individual specific features from spectrograms. Last, we build the WifiU system that can recognize humans based on the extracted features. Acknowledgments This work is partially supported by the National Natural Science Foundation of China under Grant Numbers 61373129, 61472184, 61321491, and 61472185, the National Science Foundation under Grant Numbers CNS-1421407, Collaborative Innovation Center of Novel Software Technology and Industrialization, and the Jiangsu High-level Innovation and Entrepreneurship (Shuangchuang) Program. REFERENCES 1. Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015a. Capturing the human figure through a wall. ACM Transactions on Graphics 34, 6 (2015), 219
