Unlock with Your Heart:Heartbeat-based Authentication on Commercial Mobile Phones LEI WANG,State Key Laboratory for Novel Software Technology,Nanjing University,China KANG HUANG,State Key Laboratory for Novel Software Technology,Nanjing University,China KE SUN,State Key Laboratory for Novel Software Technology,Nanjing University,China WEI WANG,State Key Laboratory for Novel Software Technology,Nanjing University,China CHEN TIAN,State Key Laboratory for Novel Software Technology,Nanjing University,China LEI XIE,State Key Laboratory for Novel Software Technology,Nanjing University,China QING GU,State Key Laboratory for Novel Software Technology,Nanjing University,China In this paper,we propose to use the vibration of the chest in response to the heartbeat as a biometric feature to authenticate the user on mobile devices.We use the built-in accelerometer to capture the heartbeat signals on commercial mobile phones. The user only needs to press the phone on his/her chest,and the system can identify the user within a few heartbeats.To reliably extract heartbeat features,we design a two-step alignment scheme that can handle the natural variability in human heart rates.We further use an adaptive template selection scheme to authenticate the user under different body postures and body states.Based on heartbeat signals collected on twenty users,the experimental results show that our method can achieve an authentication accuracy of 96.49%and the heartbeat features are stable over a period of three months. CCS Concepts:.Security and privacy-Biometrics; Additional Key Words and Phrases:Biometrics-based Authentication,Mobile System ACM Reference Format: Lei Wang,Kang Huang.Ke Sun,Wei Wang.Chen Tian,Lei Xie,and Qing Gu.2018.Unlock with Your Heart:Heartbeat- based Authentication on Commercial Mobile Phones.Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.2,3,Article 140 (September 2018),22 pages.https://doiorg/10.1145/3264950 1 INTRODUCTION Biometric features,including fingerprints and faces,have been used as metrics for user authentication on commercial mobile devices.Biometrics-based user authentication systems identify the user based on "who you are",instead of "what you know"(passwords)or "what you have"(tokens)[47].Since users often forget to carry their physical tokens and passwords are susceptible to leakage [5,54],biometrics-based authentication systems Authors'addresses:Lei Wang,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing,Jiangsu,China,wangl@smail nju.edu.cn:Kang Huang.State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing.Jiangsu,China,hkwany520@ gmail.com;Ke Sun,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing.Jiangsu,China,kesun@smail.nju.edu.cn Wei Wang.State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing.Jiangsu,China,ww@njuedu.cn;Chen Tian, State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing.Jiangsu,China,tianchen@nju.edu.cn;Lei Xie,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing,Jiangsu,China,Ixie@nju.edu.cn;Qing Gu,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing.Jiangsu,China,guq@nju.edu.cn Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.Copyrights for components of this work owned by others than ACM must be honored.Abstracting with credit is permitted.To copy otherwise,or republish,to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.Request permissions from permissions@acm.org. 2018 Association for Computing Machinery. 2474-9567/2018/9-ART140$15.00 https:/doi.org/10.1145/3264950 Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018. 140
140 Unlock with Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones LEI WANG, State Key Laboratory for Novel Software Technology, Nanjing University, China KANG HUANG, State Key Laboratory for Novel Software Technology, Nanjing University, China KE SUN, State Key Laboratory for Novel Software Technology, Nanjing University, China WEI WANG, State Key Laboratory for Novel Software Technology, Nanjing University, China CHEN TIAN, State Key Laboratory for Novel Software Technology, Nanjing University, China LEI XIE, State Key Laboratory for Novel Software Technology, Nanjing University, China QING GU, State Key Laboratory for Novel Software Technology, Nanjing University, China In this paper, we propose to use the vibration of the chest in response to the heartbeat as a biometric feature to authenticate the user on mobile devices. We use the built-in accelerometer to capture the heartbeat signals on commercial mobile phones. The user only needs to press the phone on his/her chest, and the system can identify the user within a few heartbeats. To reliably extract heartbeat features, we design a two-step alignment scheme that can handle the natural variability in human heart rates. We further use an adaptive template selection scheme to authenticate the user under different body postures and body states. Based on heartbeat signals collected on twenty users, the experimental results show that our method can achieve an authentication accuracy of 96.49% and the heartbeat features are stable over a period of three months. CCS Concepts: • Security and privacy → Biometrics; Additional Key Words and Phrases: Biometrics-based Authentication, Mobile System ACM Reference Format: Lei Wang, Kang Huang, Ke Sun, Wei Wang, Chen Tian, Lei Xie, and Qing Gu. 2018. Unlock with Your Heart: Heartbeatbased Authentication on Commercial Mobile Phones. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 3, Article 140 (September 2018), 22 pages. https://doi.org/10.1145/3264950 1 INTRODUCTION Biometric features, including fingerprints and faces, have been used as metrics for user authentication on commercial mobile devices. Biometrics-based user authentication systems identify the user based on “who you are”, instead of “what you know” (passwords) or “what you have” (tokens) [47]. Since users often forget to carry their physical tokens and passwords are susceptible to leakage [5, 54], biometrics-based authentication systems Authors’ addresses: Lei Wang, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, wangl@smail. nju.edu.cn; Kang Huang, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, hkwany520@ gmail.com; Ke Sun, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, kesun@smail.nju.edu.cn; Wei Wang, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, ww@nju.edu.cn; Chen Tian, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, tianchen@nju.edu.cn; Lei Xie, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, lxie@nju.edu.cn; Qing Gu, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, Jiangsu, China, guq@nju.edu.cn. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2018 Association for Computing Machinery. 2474-9567/2018/9-ART140 $15.00 https://doi.org/10.1145/3264950 Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
140:2·L.Wang et al. provide a convenient and secure way to unlock private mobile devices,i.e.,devices that often have a singular user, including smartphones and smartwatches.However,most biometric features,such as fingerprints,faces,and voices,are vulnerable to spoofing and replaying attacks [4,13,17,43].For example,with the widely available 3D-reconstruction and 3D-printing technologies,it is easy to bypass face recognition systems with 3D masks [17]. Therefore,we need to find a new biometric feature that is easily accessible on mobile devices and yet difficult to be reproduced by attackers. The vibration of the chest in response to the heartbeat,which is called seismocardiogram(SCG)[26],can be used as a biometric feature for user authentication.Firstly,the heartbeat pattern depends on the biological features and geometric structure of the heart,which is unique for each person.Secondly,SCG provides strong protection against spoofing attacks.To access the SCG,the adversaries have to attach a device to the chest of the user,which is considerably harder than taking photos of the user's face or recording the voices of the user. While there are contactless radar systems that can measure the heartbeat from a distance [39,66],there is still no evidence that these signals are reliable enough for reconstructing the details of heartbeat dynamics.Furthermore, compared to replaying the heartbeat sound,it is harder for the adversaries to reproduce the small vibrations caused by heartbeats.Thirdly,the heartbeat pattern is closely linked to the"liveness"and the emotion of the user.By detecting the abnormality of the heartbeat pattern,the system can potentially reject the user when he/she is under threat.While SCG can serve as the biometric feature for user authentication,traditional SCG measurement schemes require specially designed devices and need to attach the device via chest bands [26].This makes traditional SCG approaches not applicable to authentication on commercial mobile devices. In this paper,we propose to use the built-in accelerometer to capture the heartbeat vibration and perform user authentication on commercial mobile devices.To unlock the device,the user only needs to press the device on his/her chest to collect heartbeat signals,and the system can identify the user within a few heartbeats,as shown in Figure 1(a).Our design is based on the observation that the detailed vibration patterns within one heartbeat cycle can serve as a unique identity for a person,and such patterns can be reliably captured by the accelerometers of commercial mobile phones.Using SCG collected from twenty volunteers,we find that different people have different heartbeat patterns even if their heart rates are similar.Moreover,these patterns are robust when the user slightly changes the position where the heartbeat is measured or the angle of the mobile phone.Therefore,this authentication scheme can be easily used in daily life.Heartbeat patterns can serve as the main authentication scheme for mobile devices,or as a supplementary authentication scheme in multi-factor authentication solutions For example,a two-factor authentication system may ask the user to press the phone on his/her chest and put one finger on the fingerprint scanner at the same time.In this way,the system checks both the fingerprint and the heartbeat pattern to improve the security level of the authentication process. When building heartbeat-based authentication system,we need to address the following technical challenges. First,human heartbeat patterns contain intrinsic Heart Rate Variability(HRV)[42].Even for a healthy person, the standard deviation of the time between two normal heartbeats(SDNN)could be larger than one hundred milliseconds(one-tenth of the heartbeat cycle).This is because heartbeats are susceptible to variations in the inputs from the parasympathetic nervous system(PSNS)caused by multiple human factors,e.g.,respiration.The variability in heartbeat duration leads to challenges in dividing and aligning the heartbeat signals.To address this challenge,we propose a two-step segmentation and alignment scheme that can precisely align the key timing features of the heartbeat even if the durations of the heartbeats are slightly different.Second,extracting reliable features from heartbeat signals with different durations is challenging.On one hand,the heartbeat signals from different persons contain similar peak-and-valley sequences with slightly different amplitudes and time intervals. On the other hand,directly using the raw heartbeat signal and matching in the time domain often wrongly reject the authorized user due to the variation in the duration of a heartbeat cycle.To address this challenge,we propose to use wavelet transform to extract features from heartbeat signals.Our experimental results show that features extracted by wavelet transform outperform both the Dynamic Time Warping(DTW)and time domain matching Proc.ACM Interact.Mob.Wearable Ubiquitous Technol..Vol 2.No.3.Article 140.Publication date:September 2018
140:2 • L. Wang et al. provide a convenient and secure way to unlock private mobile devices, i.e., devices that often have a singular user, including smartphones and smartwatches. However, most biometric features, such as fingerprints, faces, and voices, are vulnerable to spoofing and replaying attacks [4, 13, 17, 43]. For example, with the widely available 3D-reconstruction and 3D-printing technologies, it is easy to bypass face recognition systems with 3D masks [17]. Therefore, we need to find a new biometric feature that is easily accessible on mobile devices and yet difficult to be reproduced by attackers. The vibration of the chest in response to the heartbeat, which is called seismocardiogram (SCG) [26], can be used as a biometric feature for user authentication. Firstly, the heartbeat pattern depends on the biological features and geometric structure of the heart, which is unique for each person. Secondly, SCG provides strong protection against spoofing attacks. To access the SCG, the adversaries have to attach a device to the chest of the user, which is considerably harder than taking photos of the user’s face or recording the voices of the user. While there are contactless radar systems that can measure the heartbeat from a distance [39, 66], there is still no evidence that these signals are reliable enough for reconstructing the details of heartbeat dynamics. Furthermore, compared to replaying the heartbeat sound, it is harder for the adversaries to reproduce the small vibrations caused by heartbeats. Thirdly, the heartbeat pattern is closely linked to the “liveness” and the emotion of the user. By detecting the abnormality of the heartbeat pattern, the system can potentially reject the user when he/she is under threat. While SCG can serve as the biometric feature for user authentication, traditional SCG measurement schemes require specially designed devices and need to attach the device via chest bands [26]. This makes traditional SCG approaches not applicable to authentication on commercial mobile devices. In this paper, we propose to use the built-in accelerometer to capture the heartbeat vibration and perform user authentication on commercial mobile devices. To unlock the device, the user only needs to press the device on his/her chest to collect heartbeat signals, and the system can identify the user within a few heartbeats, as shown in Figure 1(a). Our design is based on the observation that the detailed vibration patterns within one heartbeat cycle can serve as a unique identity for a person, and such patterns can be reliably captured by the accelerometers of commercial mobile phones. Using SCG collected from twenty volunteers, we find that different people have different heartbeat patterns even if their heart rates are similar. Moreover, these patterns are robust when the user slightly changes the position where the heartbeat is measured or the angle of the mobile phone. Therefore, this authentication scheme can be easily used in daily life. Heartbeat patterns can serve as the main authentication scheme for mobile devices, or as a supplementary authentication scheme in multi-factor authentication solutions. For example, a two-factor authentication system may ask the user to press the phone on his/her chest and put one finger on the fingerprint scanner at the same time. In this way, the system checks both the fingerprint and the heartbeat pattern to improve the security level of the authentication process. When building heartbeat-based authentication system, we need to address the following technical challenges. First, human heartbeat patterns contain intrinsic Heart Rate Variability (HRV) [42]. Even for a healthy person, the standard deviation of the time between two normal heartbeats (SDNN) could be larger than one hundred milliseconds (one-tenth of the heartbeat cycle). This is because heartbeats are susceptible to variations in the inputs from the parasympathetic nervous system (PSNS) caused by multiple human factors, e.g., respiration. The variability in heartbeat duration leads to challenges in dividing and aligning the heartbeat signals. To address this challenge, we propose a two-step segmentation and alignment scheme that can precisely align the key timing features of the heartbeat even if the durations of the heartbeats are slightly different. Second, extracting reliable features from heartbeat signals with different durations is challenging. On one hand, the heartbeat signals from different persons contain similar peak-and-valley sequences with slightly different amplitudes and time intervals. On the other hand, directly using the raw heartbeat signal and matching in the time domain often wrongly reject the authorized user due to the variation in the duration of a heartbeat cycle. To address this challenge, we propose to use wavelet transform to extract features from heartbeat signals. Our experimental results show that features extracted by wavelet transform outperform both the Dynamic Time Warping (DTW) and time domain matching Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
Unlock with Your Heart:Heartbeat-based Authentication on Commercial Mobile Phones.140:3 Unlock With Your Heart Yes! (a)Capturing the SCG signals using the mo- (b)Interface of our Android APp bile phone Fig.1.Heartbeat-based authentication scenario schemes.Third,human heartbeat patterns change under various conditions.For example,the heartbeat patterns captured after exercising are different to the pattern when the same user is in the resting state,even if these patterns are normalized in the time domain so that their heartbeat cycles are stretched to the same duration.To address this challenge,we propose a heartbeat pattern selection scheme that chooses the best heartbeat patterns for authentication based on the scenario information,which indicates the status of the user(e.g.,whether the user is in the exercising or the resting state)and the body posture(e.g.whether the user is standing/sitting.lying down or leaning on the sofa). We have implemented our heartbeat-based authentication scheme on the Android platform.We collected more than 110,000 heartbeat samples from 35 volunteers.The implemented system achieves an Equal Error Rate(EER) of 3.51%for user authentication when using just five heartbeat cycles.Our experimental results also show that the system is robust against different ways of putting the mobile phone and different body postures. In summary,we have made the following contributions in this paper: To our best knowledge,we are the first to perform heartbeat-based user authentication using the built-in accelerometer on commercial mobile phones. We propose a set of novel signal processing schemes designed for heartbeat-based user authentication, including template-based heartbeat alignment,wavelet-based feature extraction,and dynamic heartbeat pattern selection We implement our authentication system on commercial smartphones and verify our design using heartbeat signals collected from twenty users. 2 RELATED WORK Existing work on heartbeat measurement and authentication can be divided into three categories:special equipment based heartbeat measurement,commodity device based heartbeat measurement,and biometrics-based authentication. Special Equipment based Heartbeat Measurement:Existing systems use specialized equipment to collect heartbeat signals,including electrocardiography(ECG),ballistocardiogram(BCG),seismocardiogram(SCG)and RF cardiac signals.ECG signal has been used for heart rate estimation [46,58]and disease diagnosis [33,37] for a long time.While ECG provides accurate heartbeat measurements,ECG systems have to attach electrodes Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2.No.3,Article 140.Publication date:September 2018
Unlock with Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones • 140:3 Z - a xis X-axis Y-axis (a) Capturing the SCG signals using the mobile phone (b) Interface of our Android APP Fig. 1. Heartbeat-based authentication scenario schemes. Third, human heartbeat patterns change under various conditions. For example, the heartbeat patterns captured after exercising are different to the pattern when the same user is in the resting state, even if these patterns are normalized in the time domain so that their heartbeat cycles are stretched to the same duration. To address this challenge, we propose a heartbeat pattern selection scheme that chooses the best heartbeat patterns for authentication based on the scenario information, which indicates the status of the user (e.g., whether the user is in the exercising or the resting state) and the body posture (e.g., whether the user is standing/sitting, lying down or leaning on the sofa). We have implemented our heartbeat-based authentication scheme on the Android platform. We collected more than 110, 000 heartbeat samples from 35 volunteers. The implemented system achieves an Equal Error Rate (EER) of 3.51% for user authentication when using just five heartbeat cycles. Our experimental results also show that the system is robust against different ways of putting the mobile phone and different body postures. In summary, we have made the following contributions in this paper: • To our best knowledge, we are the first to perform heartbeat-based user authentication using the built-in accelerometer on commercial mobile phones. • We propose a set of novel signal processing schemes designed for heartbeat-based user authentication, including template-based heartbeat alignment, wavelet-based feature extraction, and dynamic heartbeat pattern selection. • We implement our authentication system on commercial smartphones and verify our design using heartbeat signals collected from twenty users. 2 RELATED WORK Existing work on heartbeat measurement and authentication can be divided into three categories: special equipment based heartbeat measurement, commodity device based heartbeat measurement, and biometrics-based authentication. Special Equipment based Heartbeat Measurement: Existing systems use specialized equipment to collect heartbeat signals, including electrocardiography (ECG), ballistocardiogram (BCG), seismocardiogram (SCG) and RF cardiac signals. ECG signal has been used for heart rate estimation [46, 58] and disease diagnosis [33, 37] for a long time. While ECG provides accurate heartbeat measurements, ECG systems have to attach electrodes Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
140:4·L.Wang et al. to the skin of the user,which is inconvenient for daily use.BCG measures the micro recoil movements of the body caused by the blood traveling along the vascular tree [8,20,41].Such micro-movements can be captured by highly sensitive geophone mounted on the bed that the user is sleeping on [29,30].SCG measures the local vibration of the chest caused by the heartbeat and it has been used for heart rate estimation [9,36,52,56].SCG can also be used for assessments of the time interval of different mechanical events occurring during the systolic and diastolic phase [14-16].However,most SCG systems require specifically designed chest belt to attach the sensor to the chest of the user [14].Recently,RF-based systems provide a non-intrusive and contactless way for heartbeat measurement.Adib et al.[1]use Frequency Modulated Continuous Wave(FMCW)to monitor the heart rates with a median accuracy of 99%.Yang et al.[64]propose a system that uses 60GHz millimeter wave (mmWave)for heartbeat monitoring.However,most of these systems use expensive special hardware and only provide coarse heart rate estimations that are not applicable for user authentication. Commodity Device based Heartbeat Measurement:Low-cost commodity devices,including Wi-Fi devices and smartphones,can also be used for heartbeat monitoring.With the Channel State Information(CSI)captured from commercial WI-Fi devices,it is possible to estimate the heart rate by either the amplitude of CSI [40]or the phase of CSI [63].Furthermore,Zhao et al.[66]show that CSI provides enough details in heartbeat cycles so that it can be used for recognizing the emotional state of the user.Oian et al.[51]leverage inaudible acoustic signals emitted by commodity mobile phones to monitor the heart rates.However,these Wi-Fi and acoustic signal based measurements are sensitive to environmental changes,including the angle and the distance of the device to the target user. There are systems that use the built-in accelerometers or gyroscopes in commodity mobile phone to capture the SCG signals [35,44,59].Most of these systems only provide coarse measurements,such as heart rates or Heart Rate Variability(HRV)[35,44].In a recent system deployed on smartphones,Wang et al.[59]detect the detailed fiducial point of the SCG signals with the aid of photoplethysmogram(PPG)to measure the blood pressure of the user.In comparison,our system solely relies on the SCG signals captured by the built-in accelerometer to extract detailed heart movement pattern without help from other sensors. Biometrics based Authentication:Biometrics-based authentication uses features,such as fingerprint [53,55], face [18,21],voice [7,19,31,49],breath [11],iris [57],and heartbeat [12,24],to authenticate the user.Among these features,the heartbeat pattern is a relatively new and hard-to-spoof biometric feature for authentication. Choudhary and Manikandan [12]propose a heartbeat extraction framework for authentication based on ECG signals.BreathLive [24]uses a heartbeat sound based authentication system,which relies on the inherent correlation between chest motion and sounds caused by deep respiration to protect the user from replay attacks Auth'n'Scan [23]uses physiological information,including heart rates,HRV,and respiration rates,derived from PPG to authenticate the user.Cardiac Scan [39]uses a remote,high-resolution heartbeat monitoring system based on DC-coupled continuous-wave radar to achieve continuous user authentication.However,most of these heartbeat-based authentication systems use specially designed equipment and cannot be easily applied to current commodity mobile devices. 3 SYSTEM OVERVIEW 3.1 Authentication Model and System Components Our heartbeat-based authentication system aims at identifying the owner of the mobile device.We assume that the mobile device only has one owner.However,our system can be extended to identify multiple users on the same device by updating our training and recognition process. The first step of our system is the training process as shown in Figure 2.During the training process,the user needs to press the mobile device on his/her chest,more specifically,put the bottom of the phone perpendicularly on the lower portion of the sternum,to collect training heartbeat samples,as shown in Figure 1(a).The training Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018
140:4 • L. Wang et al. to the skin of the user, which is inconvenient for daily use. BCG measures the micro recoil movements of the body caused by the blood traveling along the vascular tree [8, 20, 41]. Such micro-movements can be captured by highly sensitive geophone mounted on the bed that the user is sleeping on [29, 30]. SCG measures the local vibration of the chest caused by the heartbeat and it has been used for heart rate estimation [9, 36, 52, 56]. SCG can also be used for assessments of the time interval of different mechanical events occurring during the systolic and diastolic phase [14–16]. However, most SCG systems require specifically designed chest belt to attach the sensor to the chest of the user [14]. Recently, RF-based systems provide a non-intrusive and contactless way for heartbeat measurement. Adib et al. [1] use Frequency Modulated Continuous Wave (FMCW) to monitor the heart rates with a median accuracy of 99%. Yang et al. [64] propose a system that uses 60GHz millimeter wave (mmWave) for heartbeat monitoring. However, most of these systems use expensive special hardware and only provide coarse heart rate estimations that are not applicable for user authentication. Commodity Device based Heartbeat Measurement: Low-cost commodity devices, including Wi-Fi devices and smartphones, can also be used for heartbeat monitoring. With the Channel State Information (CSI) captured from commercial WI-Fi devices, it is possible to estimate the heart rate by either the amplitude of CSI [40] or the phase of CSI [63]. Furthermore, Zhao et al. [66] show that CSI provides enough details in heartbeat cycles so that it can be used for recognizing the emotional state of the user. Qian et al. [51] leverage inaudible acoustic signals emitted by commodity mobile phones to monitor the heart rates. However, these Wi-Fi and acoustic signal based measurements are sensitive to environmental changes, including the angle and the distance of the device to the target user. There are systems that use the built-in accelerometers or gyroscopes in commodity mobile phone to capture the SCG signals [35, 44, 59]. Most of these systems only provide coarse measurements, such as heart rates or Heart Rate Variability (HRV) [35, 44]. In a recent system deployed on smartphones, Wang et al. [59] detect the detailed fiducial point of the SCG signals with the aid of photoplethysmogram (PPG) to measure the blood pressure of the user. In comparison, our system solely relies on the SCG signals captured by the built-in accelerometer to extract detailed heart movement pattern without help from other sensors. Biometrics based Authentication: Biometrics-based authentication uses features, such as fingerprint [53, 55], face [18, 21], voice [7, 19, 31, 49], breath [11], iris [57], and heartbeat [12, 24], to authenticate the user. Among these features, the heartbeat pattern is a relatively new and hard-to-spoof biometric feature for authentication. Choudhary and Manikandan [12] propose a heartbeat extraction framework for authentication based on ECG signals. BreathLive [24] uses a heartbeat sound based authentication system, which relies on the inherent correlation between chest motion and sounds caused by deep respiration to protect the user from replay attacks. Auth’n’Scan [23] uses physiological information, including heart rates, HRV, and respiration rates, derived from PPG to authenticate the user. Cardiac Scan [39] uses a remote, high-resolution heartbeat monitoring system based on DC-coupled continuous-wave radar to achieve continuous user authentication. However, most of these heartbeat-based authentication systems use specially designed equipment and cannot be easily applied to current commodity mobile devices. 3 SYSTEM OVERVIEW 3.1 Authentication Model and System Components Our heartbeat-based authentication system aims at identifying the owner of the mobile device. We assume that the mobile device only has one owner. However, our system can be extended to identify multiple users on the same device by updating our training and recognition process. The first step of our system is the training process as shown in Figure 2. During the training process, the user needs to press the mobile device on his/her chest, more specifically, put the bottom of the phone perpendicularly on the lower portion of the sternum, to collect training heartbeat samples, as shown in Figure 1(a). The training Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
Unlock with Your Heart:Heartbeat-based Authentication on Commercial Mobile Phones.140:5 AuthenticatingHeartbeat Feature User Extract Scenario Infomation Segmentation Alignment Extraction Authentication Heartbeat Heart Rate Fine Template SVM Model Collection Estimation Generate Feature SVM Model Fine Alianmen Extractior Training Template Generation Fig.2.Authentication System Components process normally takes less than two minutes(for collecting 60 heartbeats).Users may be instructed to change the position or the angle of the device during the training process to introduce more variations in the training samples When collecting the training samples,our system records the built-in accelerometer readings at a sampling rate of 100~250 Hz(depending on the hardware support of the device).With the readings of the accelerometer,we first extract the heart rates and the body posture of the user.With this information,the collected training samples can be classified into one of the predefined scenarios,e.g,the heart rates are in the range of 50~80 Beats per Minute(BPM)and the user is sitting on a chair.The training samples are then used for generating heartbeat patterns for that given scenario.Each heartbeat pattern includes one heartbeat template for signal alignment and one Support Vector Machine(SVM)model for identifying the owner of the device.The SVM model is a two-class classifier that is trained using the training heartbeats from the owner(as the positive samples)and the benchmark heartbeats from a global heartbeat database(as the negative samples).The SVM model can give the likelihood whether an unknown heartbeat signal belongs to the owner or not. After the training process,our system uses the heartbeat patterns to perform user authentication.Similar to the training process,the authentication process first collects the heartbeat signals and then extracts the scenario information from the readings of the accelerometer.The scenario information is used for selecting one set of the heartbeat patterns,including both the template for signal alignment and the SVM model for authentication.If there is a matching heartbeat pattern in the database,the system first uses the template to segment the continuous SCG signals into individual heartbeat cycles and align the key features of each cycle.The system then extracts features using wavelet transform and applies the SVM model to classify the heartbeats.If there is no heartbeat pattern for the identified scenario,the system fallbacks to another authentication scheme,such as asking the user to input a PIN.If the user is authenticated through the PIN,the buffered heartbeat signals are used for generating the new heartbeat pattern(both the alignment template and the SVM model)for the identified scenario. The key components of our system are described in the following sections: Heartbeat Segmentation and Alignment(Section 4):In the heartbeat segmentation component,we use a two-step segmentation algorithm to divide the continuous acceleration signals into individual heartbeat cycles. The first step is coarse heart rate estimation,which uses a coarse template to estimate the heart rates from the accelerometer readings.The estimated heart rates are used for selecting the heartbeat pattern which contains the template for fine-grained heartbeat alignment.In the second step of heartbeat segmentation,we use the fine template to perform a cross-correlation on the continuous heartbeat signals.By this way,we can precisely align the key features of each heartbeat cycle in the time domain. Feature Extraction(Section 5):After the segmentation step,our system performs data preprocessing,e.g., normalizing the amplitude of the heartbeat signals,before the feature extraction step.Then,we use Discrete Wavelet Transform(DWT)to extract features from the heartbeat.Each heartbeat cycle is decomposed into multiple levels of wavelet coefficients,and we choose the wavelet coefficients that are most closely related to the heartbeat patterns.This way,we reduce noises that come from different sources,including the respiration movements,small limb movements,and small variations in accelerometer readings. Proc.ACM Interact.Mob.Wearable Ubiquitous Technol..Vol.2.No.3.Article 140.Publication date:September 2018
Unlock with Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones • 140:5 Heartbeat Collection Training Extract Scenario Infomation Heart Rate Estimation Posture Estimation Scenario Selection Authenticating Heartbeat Segmentation & Alignment Generate Fine Alignment Template Feature Extraction User Authentication Feature Extraction SVM Model Generation Fine Template SVM Model Fig. 2. Authentication System Components process normally takes less than two minutes (for collecting 60 heartbeats). Users may be instructed to change the position or the angle of the device during the training process to introduce more variations in the training samples. When collecting the training samples, our system records the built-in accelerometer readings at a sampling rate of 100∼250 Hz (depending on the hardware support of the device). With the readings of the accelerometer, we first extract the heart rates and the body posture of the user. With this information, the collected training samples can be classified into one of the predefined scenarios, e.g., the heart rates are in the range of 50 ∼ 80 Beats per Minute (BPM) and the user is sitting on a chair. The training samples are then used for generating heartbeat patterns for that given scenario. Each heartbeat pattern includes one heartbeat template for signal alignment and one Support Vector Machine (SVM) model for identifying the owner of the device. The SVM model is a two-class classifier that is trained using the training heartbeats from the owner (as the positive samples) and the benchmark heartbeats from a global heartbeat database (as the negative samples). The SVM model can give the likelihood whether an unknown heartbeat signal belongs to the owner or not. After the training process, our system uses the heartbeat patterns to perform user authentication. Similar to the training process, the authentication process first collects the heartbeat signals and then extracts the scenario information from the readings of the accelerometer. The scenario information is used for selecting one set of the heartbeat patterns, including both the template for signal alignment and the SVM model for authentication. If there is a matching heartbeat pattern in the database, the system first uses the template to segment the continuous SCG signals into individual heartbeat cycles and align the key features of each cycle. The system then extracts features using wavelet transform and applies the SVM model to classify the heartbeats. If there is no heartbeat pattern for the identified scenario, the system fallbacks to another authentication scheme, such as asking the user to input a PIN. If the user is authenticated through the PIN, the buffered heartbeat signals are used for generating the new heartbeat pattern (both the alignment template and the SVM model) for the identified scenario. The key components of our system are described in the following sections: Heartbeat Segmentation and Alignment (Section 4): In the heartbeat segmentation component, we use a two-step segmentation algorithm to divide the continuous acceleration signals into individual heartbeat cycles. The first step is coarse heart rate estimation, which uses a coarse template to estimate the heart rates from the accelerometer readings. The estimated heart rates are used for selecting the heartbeat pattern which contains the template for fine-grained heartbeat alignment. In the second step of heartbeat segmentation, we use the fine template to perform a cross-correlation on the continuous heartbeat signals. By this way, we can precisely align the key features of each heartbeat cycle in the time domain. Feature Extraction (Section 5): After the segmentation step, our system performs data preprocessing, e.g., normalizing the amplitude of the heartbeat signals, before the feature extraction step. Then, we use Discrete Wavelet Transform (DWT) to extract features from the heartbeat. Each heartbeat cycle is decomposed into multiple levels of wavelet coefficients, and we choose the wavelet coefficients that are most closely related to the heartbeat patterns. This way, we reduce noises that come from different sources, including the respiration movements, small limb movements, and small variations in accelerometer readings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
140:6·L.Wang et al,. ATC Heartbeat Cycle -0.1 0.2 03 0.20.40.60.8 0 0.20.40.6 0.8 Time (s) Time (s) (a)Heartbeat motion stages (b)Heartbeat pattern of volunteer A (c)Heartbeat pattern of volunteer B Fig.3.Heartbeat movement cycle and pattern Time (s) Time(s) (a)Volunteer A (b)Volunteer B Fig.4.Five consecutive heartbeat cycles for volunteer A and B User Authentication(Section 6):Heartbeat authentication uses the SVM model for the heartbeat pattern of the given scenario.We first perform a per-heartbeat evaluation that gives the likelihood that the given heartbeat is from the authorized user.We then combine the likelihood of multiple consecutive heartbeats to improve the confidence in the decision.Our system dynamically determines the number of heartbeats that are required for the authentication process.For example,if the heartbeats have a consistently high likelihood of belonging to the authorized user,the authentication may only require as few as five heartbeats.If the system is not confident in the decision,it may instruct the user to press the phone on the chest for a longer time so that more heartbeat samples can be collected to improve the confidence. 3.2 Background of the SCG Signal The seismocardiogram(SCG)signals collected by accelerometers capture the heartbeat motion of the user. Heartbeat motion is a 3D self-driving heart deformation arising from the stimulation of the cardiac muscle [22].The human heart has two upper chambers(i.e.atria)and two bottom chambers(i.e.ventricles)[32].The continuous contraction and relaxation of atria and ventricles cause the heartbeat motion.As shown in Fig.3(a). one heartbeat motion cycle consists of seven stages:(1)atrial contraction(ATC),(2)mitral valve closing(MC).(3) aortic valve opening(AO),(4)point of maximal acceleration in the aorta (MA),(5)aortic valve closure (AC),(6) mitral valve opening(MO),(7)rapid filling of left ventricle (RF)[16,25]. The motion stages of the heartbeat cycle can be captured and identified using the accelerometer readings provided by mobile phones,see Figure 3(b).As the phone is pressed perpendicularly on the chest,we always use the readings of the y-axis of the accelerometer(pointing from the bottom to the top of the phone).Depending on the stage of the heartbeat cycle,the acceleration caused by the heart motion could be positive or negative. Therefore,each stage in the heartbeat cycle corresponds to one of the peaks or valleys in the SCG signal.Based on our measurements,the average amplitude of the AO peak is 0.2558 m/s2(SD=0.0384 m/s2).The background noise level of the accelerometer has a variance of 0.0104 m/s2.Therefore,commercial mobile phones provide enough Signal-to-Noise Ratio(SNR)for measuring the details in SCG signals. Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018
140:6 • L. Wang et al. ATC MC AO MA AC MO RF Heartbeat Cycle (a) Heartbeat motion stages 0 0.2 0.4 0.6 0.8 Time (s) -0.5 0 0.5 Acceleration (m/s 2 ) ATC MC MA AO MO AC RF (b) Heartbeat pattern of volunteer A 0 0.2 0.4 0.6 0.8 Time (s) -0.3 -0.2 -0.1 0 0.1 0.2 Acceleration (m/s 2 ) ATC MC AO MA RF MO AC (c) Heartbeat pattern of volunteer B Fig. 3. Heartbeat movement cycle and pattern 01234 Time (s) -0.5 0 0.5 Acceleration (m/s 2 ) AO RF AO RF (a) Volunteer A 01234 Time (s) -0.2 0 0.2 AO RF AO RF Acceleration (m/s 2 ) (b) Volunteer B Fig. 4. Five consecutive heartbeat cycles for volunteer A and B User Authentication (Section 6): Heartbeat authentication uses the SVM model for the heartbeat pattern of the given scenario. We first perform a per-heartbeat evaluation that gives the likelihood that the given heartbeat is from the authorized user. We then combine the likelihood of multiple consecutive heartbeats to improve the confidence in the decision. Our system dynamically determines the number of heartbeats that are required for the authentication process. For example, if the heartbeats have a consistently high likelihood of belonging to the authorized user, the authentication may only require as few as five heartbeats. If the system is not confident in the decision, it may instruct the user to press the phone on the chest for a longer time so that more heartbeat samples can be collected to improve the confidence. 3.2 Background of the SCG Signal The seismocardiogram (SCG) signals collected by accelerometers capture the heartbeat motion of the user. Heartbeat motion is a 3D self-driving heart deformation arising from the stimulation of the cardiac muscle [22]. The human heart has two upper chambers (i.e. atria) and two bottom chambers (i.e. ventricles) [32]. The continuous contraction and relaxation of atria and ventricles cause the heartbeat motion. As shown in Fig.3(a), one heartbeat motion cycle consists of seven stages: (1) atrial contraction (ATC), (2) mitral valve closing (MC), (3) aortic valve opening (AO), (4) point of maximal acceleration in the aorta (MA), (5) aortic valve closure (AC), (6) mitral valve opening (MO), (7) rapid filling of left ventricle (RF) [16, 25]. The motion stages of the heartbeat cycle can be captured and identified using the accelerometer readings provided by mobile phones, see Figure 3(b). As the phone is pressed perpendicularly on the chest, we always use the readings of the y-axis of the accelerometer (pointing from the bottom to the top of the phone). Depending on the stage of the heartbeat cycle, the acceleration caused by the heart motion could be positive or negative. Therefore, each stage in the heartbeat cycle corresponds to one of the peaks or valleys in the SCG signal. Based on our measurements, the average amplitude of the AO peak is 0.2558 m/s 2 (SD= 0.0384 m/s 2 ). The background noise level of the accelerometer has a variance of 0.0104 m/s 2 . Therefore, commercial mobile phones provide enough Signal-to-Noise Ratio (SNR) for measuring the details in SCG signals. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
Unlock with Your Heart:Heartbeat-based Authentication on Commercial Mobile Phones.140:7 0.012 2.5 0.05 -Volunteer A -Volunteer A -Volunteer A 0.01 Volunteer B Volunteer E 0.04 Volunteer B Volunteer C Volunteer C Volunteer C 密0.008 Volunteer D Volunteer D Volunteer D 0.006 Volunt rE Volunteer E Voluntee g0.04 0.02 0.002 在0.01 .400 -200 0 200 400 3 -50 0 50 Deviation(ms) Ratio (AO/RF) Deviation(ms) (a)Deviation of heartbeat interval from the (b)Ratio of AO amplitude to RF Amplitude (c)Deviation of AO-RF interval from the mean mean value value Fig.5.Variations in the SCG signal 3.3 Characteristics of the SCG Signal By looking at the SCG waveforms,we have the following observations that lead to the possibility of using the SCG signal for authentication: First,the SCG signals of different people go through the same seven stages,but have different signal patterns in terms of amplitudes of the corresponding peaks and intervals between peaks.Figure 3(b)and Figure 3(c)show two SCG samples of one heartbeat cycle from two volunteers.While both volunteers have similar heart rates(73 BPM and 71 BPM,respectively),the two SCG patterns have distinctive features.For example,the amplitudes of the AO peaks for the two volunteers are quite different.Such difference in heartbeat motion comes from the differences in the size,position and shape of the heart [38].Therefore,the heartbeat motion patterns contain unique biometric features of the given user [22]. Second,the SCG signals of the same user are consistent over time.Figure 4 shows five consecutive heartbeat patterns of two volunteers.While there are small variations in the signals,we observe that the heartbeat patterns from the same person are consistent for consecutive heartbeat cycles.Furthermore,with heartbeat patterns collected across three months and with different clothes,we find that heartbeat patterns of the same user are quite stable.Therefore,the SCG signal can potentially serve as a consistent identity for the user. 4 HEARTBEAT SEGMENTATION AND ALIGNMENT In this section,we describe the heartbeat segmentation and alignment process,in which the continuous heartbeat signals are divided into individual heartbeat cycles.High precision signal alignment is vital to heartbeat- based authentication systems.This is because a misaligned heartbeat signal will lead to incorrect positioning of the different heartbeat stages.Consequently,such incorrect positioning will lead to errors in user authentication. However,due to the variances in both the amplitude and timing of the SCG signals,it is challenging to precisely align the heartbeat signals. 4.1 Variations in the SCG Signal While human heartbeats are repetitive motions,ECG-based experiments show that heartbeats are not perfectly periodical [2,3,27,45].Therefore,the SCG signals also have variations in both the amplitude and timing of the peaks corresponding to different heart motion stages. First,human heartbeat rates are not stable.There are intrinsic Heartbeat Rate Variability(HRV)in SCG signals [42].Figure 5(a)shows the Probability Density Function(PDF)of the deviation in time intervals between two normal heartbeats for five volunteers sitting on the chair.The ground truth values are obtained by manually selecting the auto-correlation peaks in the SCG signals.We observe that the standard deviation of heartbeat interval is 46ms,which is consistent with results from ECG signals [42].Thus,the duration of heartbeat cycle Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2.No.3,Article 140.Publication date:September 2018
Unlock with Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones • 140:7 -400 -200 0 200 400 Deviation (ms) 0 0.002 0.004 0.006 0.008 0.01 0.012 Probability Density Volunteer A Volunteer B Volunteer C Volunteer D Volunteer E (a) Deviation of heartbeat interval from the mean value 0123 Ratio (AO/RF) 0 0.5 1 1.5 2 2.5 Probability Density Volunteer A Volunteer B Volunteer C Volunteer D Volunteer E (b) Ratio of AO amplitude to RF Amplitude -50 0 50 Deviation (ms) 0 0.01 0.02 0.03 0.04 0.05 Probability Density Volunteer A Volunteer B Volunteer C Volunteer D Volunteer E (c) Deviation of AO-RF interval from the mean value Fig. 5. Variations in the SCG signal 3.3 Characteristics of the SCG Signal By looking at the SCG waveforms, we have the following observations that lead to the possibility of using the SCG signal for authentication: First, the SCG signals of different people go through the same seven stages, but have different signal patterns in terms of amplitudes of the corresponding peaks and intervals between peaks. Figure 3(b) and Figure 3(c) show two SCG samples of one heartbeat cycle from two volunteers. While both volunteers have similar heart rates (73 BPM and 71 BPM, respectively), the two SCG patterns have distinctive features. For example, the amplitudes of the AO peaks for the two volunteers are quite different. Such difference in heartbeat motion comes from the differences in the size, position and shape of the heart [38]. Therefore, the heartbeat motion patterns contain unique biometric features of the given user [22]. Second, the SCG signals of the same user are consistent over time. Figure 4 shows five consecutive heartbeat patterns of two volunteers. While there are small variations in the signals, we observe that the heartbeat patterns from the same person are consistent for consecutive heartbeat cycles. Furthermore, with heartbeat patterns collected across three months and with different clothes, we find that heartbeat patterns of the same user are quite stable. Therefore, the SCG signal can potentially serve as a consistent identity for the user. 4 HEARTBEAT SEGMENTATION AND ALIGNMENT In this section, we describe the heartbeat segmentation and alignment process, in which the continuous heartbeat signals are divided into individual heartbeat cycles. High precision signal alignment is vital to heartbeatbased authentication systems. This is because a misaligned heartbeat signal will lead to incorrect positioning of the different heartbeat stages. Consequently, such incorrect positioning will lead to errors in user authentication. However, due to the variances in both the amplitude and timing of the SCG signals, it is challenging to precisely align the heartbeat signals. 4.1 Variations in the SCG Signal While human heartbeats are repetitive motions, ECG-based experiments show that heartbeats are not perfectly periodical [2, 3, 27, 45]. Therefore, the SCG signals also have variations in both the amplitude and timing of the peaks corresponding to different heart motion stages. First, human heartbeat rates are not stable. There are intrinsic Heartbeat Rate Variability (HRV) in SCG signals [42]. Figure 5(a) shows the Probability Density Function (PDF) of the deviation in time intervals between two normal heartbeats for five volunteers sitting on the chair. The ground truth values are obtained by manually selecting the auto-correlation peaks in the SCG signals. We observe that the standard deviation of heartbeat interval is 46ms, which is consistent with results from ECG signals [42]. Thus, the duration of heartbeat cycle Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
140:8·L.Wang et al.. Generate Coarse Template Coarse Alignment SCG Sequence Linear Locate Prune Cross Detect Heart Rate Interpolation All Peaks Noisy Peaks Correlation Correlation Peaks Fig.6.Heart Rate Estimation Scheme could be changing by as much as 1/20 of the cycle length since a normal heartbeat lasts for about one second at a heart rate of 60 BPM. Second,the peak amplitude in the SCG signal varies significantly.Figure 5(b)shows the PDF for the ratio of the amplitude of the AO peak to the RF peak in the same heartbeat cycle.These two peaks are the most prominent features in the SCG signal.Different persons have different AO to RF ratios,as shown in Figure 4.The standard deviations of AO to RF ratio is larger than 0.25 for all volunteers.This implies that the AO to RF ratio for the same person also varies significantly,e.g.in consecutive heartbeat cycles,either the AO or the RF peak could be the highest peak in the cycle,see Figure 4(b).Therefore,it is challenging to identify the AO and RF peaks using a small number of heartbeat cycles.Existing systems use hints from other measurements,such as the photoplethysmogram(PPG)[59],to help identify the AO peak.However,our system only has the SCG signals as the reference to perform the segmentation. Fortunately,we observe that the time interval between the AO stage and RF stage is relatively stable.Figure 5(c)shows the PDF of the deviation in the time interval between the AO and the RF peak.The standard deviation of the AO-RF interval is 9.48 ms,which is much smaller than that of the heartbeat interval.This implies that the ratio of the AO-RF interval to the heartbeat interval also changes significantly,as the AO-RF interval is stable and the heartbeat interval is unstable.We further verified that the AO-RF intervals are stable under different states. We collect SCG signals when users finish exercising,recline on the sofa and lie on the bed.While the heart rates are significantly higher in the exercising state,the standard deviation of the AO-RF interval is still small(i.e., 11.2 ms).The standard deviations of AO-RF interval for the reclining and lying states are 8.25 ms and 5.86 ms, respectively. Based on the above observations,we choose to use the interval between the ATC stage and the RF stage as the reference for heartbeat segmentation and alignment.We choose the ATC-RF interval due to two reasons. First,the ATC-RF interval contains the two highest peaks in the SCG signal,i.e.,AO and RF,that can be easily identified.Second,the time interval between AO and RF has smaller variations than other parts of the heartbeat cycle.We design a two-step process to divide and align the heartbeat using the signals in the reference interval as follows. 4.2 Heart Rate Estimation Given a new SCG sequence,the first step is to use a heart rate estimation algorithm,as shown in Figure 6,to measure the heart rates.To estimate the heart rates,we first use a linear interpolation algorithm to normalize the accelerometer readings to a standard sampling rate (e.g.,100 Hz).This step ensures that our system can work on mobile phones that have different sampling rates for the accelerometer. The second step of heart rate estimation is to derive a coarse-template of the reference ATC-RF interval from the SCG signals.To identify the ATC-RF interval,we first locate all the peaks(local maximum points)in an SCG sequence with a two-second duration.We assume that the heart rates of the user are between 50 BPM and 120 BPM.Therefore,there is at least one full heartbeat cycle in the two-second SCG signal.We sort the local maximum points by their amplitudes as the labels shown in Figure 7(a).We then perform a pruning algorithm to remove noisy peaks.Starting from the highest peaks,we add the peaks into a candidate set one-by-one in the descending order of their amplitudes.If the current peak is within a time interval of r to one of the candidate peaks in the set, Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018
140:8 • L. Wang et al. SCG Sequence Generate Coarse Template Linear Interpolation Prune Noisy Peaks Locate All Peaks Coarse Alignment Detect Correlation Peaks Cross Correlation Heart Rate Fig. 6. Heart Rate Estimation Scheme could be changing by as much as 1/20 of the cycle length since a normal heartbeat lasts for about one second at a heart rate of 60 BPM. Second, the peak amplitude in the SCG signal varies significantly. Figure 5(b) shows the PDF for the ratio of the amplitude of the AO peak to the RF peak in the same heartbeat cycle. These two peaks are the most prominent features in the SCG signal. Different persons have different AO to RF ratios, as shown in Figure 4. The standard deviations of AO to RF ratio is larger than 0.25 for all volunteers. This implies that the AO to RF ratio for the same person also varies significantly, e.g., in consecutive heartbeat cycles, either the AO or the RF peak could be the highest peak in the cycle, see Figure 4(b). Therefore, it is challenging to identify the AO and RF peaks using a small number of heartbeat cycles. Existing systems use hints from other measurements, such as the photoplethysmogram (PPG) [59], to help identify the AO peak. However, our system only has the SCG signals as the reference to perform the segmentation. Fortunately, we observe that the time interval between the AO stage and RF stage is relatively stable. Figure 5(c) shows the PDF of the deviation in the time interval between the AO and the RF peak. The standard deviation of the AO-RF interval is 9.48 ms, which is much smaller than that of the heartbeat interval. This implies that the ratio of the AO-RF interval to the heartbeat interval also changes significantly, as the AO-RF interval is stable and the heartbeat interval is unstable. We further verified that the AO-RF intervals are stable under different states. We collect SCG signals when users finish exercising, recline on the sofa and lie on the bed. While the heart rates are significantly higher in the exercising state, the standard deviation of the AO-RF interval is still small (i.e., 11.2 ms). The standard deviations of AO-RF interval for the reclining and lying states are 8.25 ms and 5.86 ms, respectively. Based on the above observations, we choose to use the interval between the ATC stage and the RF stage as the reference for heartbeat segmentation and alignment. We choose the ATC-RF interval due to two reasons. First, the ATC-RF interval contains the two highest peaks in the SCG signal, i.e., AO and RF, that can be easily identified. Second, the time interval between AO and RF has smaller variations than other parts of the heartbeat cycle. We design a two-step process to divide and align the heartbeat using the signals in the reference interval as follows. 4.2 Heart Rate Estimation Given a new SCG sequence, the first step is to use a heart rate estimation algorithm, as shown in Figure 6, to measure the heart rates. To estimate the heart rates, we first use a linear interpolation algorithm to normalize the accelerometer readings to a standard sampling rate (e.g., 100 Hz). This step ensures that our system can work on mobile phones that have different sampling rates for the accelerometer. The second step of heart rate estimation is to derive a coarse-template of the reference ATC-RF interval from the SCG signals. To identify the ATC-RF interval, we first locate all the peaks (local maximum points) in an SCG sequence with a two-second duration. We assume that the heart rates of the user are between 50 BPM and 120 BPM. Therefore, there is at least one full heartbeat cycle in the two-second SCG signal. We sort the local maximum points by their amplitudes as the labels shown in Figure 7(a). We then perform a pruning algorithm to remove noisy peaks. Starting from the highest peaks, we add the peaks into a candidate set one-by-one in the descending order of their amplitudes. If the current peak is within a time interval of τ to one of the candidate peaks in the set, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
Unlock with Your Heart:Heartbeat-based Authentication on Commercial Mobile Phones.140:9 1.5 6 02 0.4 0.6 Time(s) Time(s) Time(s) (a)Coarse-template selection (b)Correlation results of the coarse-template (c)Coarse alignment results Fig.7.Coarse Estimation on Heart Rate the current peak is removed.We set the threshold r to be 200 ms,as the AO-RF intervals are larger than 200 ms when the heart rate is slower than 120 BPM.After the pruning process,only the peaks corresponding to AO and RF are in the candidate set,e.g,peaks 3,14,2,11,1 and 10 in Figure 7(a).As there are multiple heartbeat cycles in the two-second SCG sequence,there are multiple candidates of AO and RF peaks.We choose the two peaks that are the closest to each other as the AO and RF for the coarse-template since the AO-RF interval is usually smaller than the RF-AO interval.We then measure the interval u between the selected AO and RF peaks.We use a segment with a duration of 1.5u as the coarse-template,starting from 0.5u before the AO peak to include the ATC stage.The resulting coarse-template is shown the blue dashed rectangle in Figure 7(a). In the third step,we perform a cross-correlation between the coarse-template with the continuous SCG signals. Figure 7(b)shows the result of the correlation,where each heartbeat corresponds to a peak that is easier to be identified than the AO or RF peaks in the raw SCG.We use a threshold based scheme to detect peaks in the correlation result.By measuring the number of correlation peaks,we can derive the heartbeat interval and the heart rates of the given SCG signal.The coarse-template based scheme gives more stable heart rate estimation than auto-correlation or FFT based schemes.This is because the similarity of AO and RF peaks for some user may introduce multiple peaks in the auto-correlation and FFT of the SCG signal,which leads to large errors in heart rate estimation. Due to the variations in the timing and amplitude of the AO and RF peaks,our coarse-template could be imprecise.For example,our heuristic algorithm could select a wrong peak to be the AO or RF.Moreover,the coarse-template derived from a single heartbeat cycle could be noisy due to the interference from breathing or other micro movements when collecting the SCG samples.Consequently,the segmentation result based on correlation of the coarse-template is not well aligned.As shown in Figure 7(c),the segmentation results of fifteen heartbeat samples collected at different times from the same person are not perfectly aligned with each other due to the errors in the coarse-template. 4.3 Fine-grained Alignment We use a fine-alignment-template to help align the SCG signals that are collected under different conditions. The fine-alignment-template is produced in the training process and we generate one fine-template for each scenario.For a new SCG sequence,we first use the heart rate estimation scheme to get the heart rates that are used for selecting the fine-template.Note that the fine-template generated in the training process can be applied to all SCG samples with a similar heart rate.Therefore,we only need to generate the fine-template once. We use an SCG sequence that contains at least ten consecutive heartbeats to generate the fine-template.First, we use the coarse segmentation scheme to divide the heartbeat signal into individual cycles.Second,we average over all the heartbeat cycles to reduce the impact of occasionally misaligned cycles and the noises caused by micro-movements.The smoothed signal is shown in Figure 8(a).Third,we use the smoothed signal to estimate the Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2.No.3,Article 140.Publication date:September 2018
Unlock with Your Heart: Heartbeat-based Authentication on Commercial Mobile Phones • 140:9 0 0.5 1 1.5 2 Time(s) -0.2 0 0.2 Acceleration (m/s 2 ) 4 5 6 7 8 9 10 15 1 11 12 14 3 2 13 (a) Coarse-template selection 0 2 4 6 8 10 Time(s) -0.3 -0.2 -0.1 0 0.1 0.2 Acceleration (m/s 2 ) (b) Correlation results of the coarse-template 0 0.2 0.4 0.6 Time (s) -0.4 -0.2 0 0.2 0.4 Acceleration (m/s 2 ) (c) Coarse alignment results Fig. 7. Coarse Estimation on Heart Rate the current peak is removed. We set the threshold τ to be 200 ms, as the AO-RF intervals are larger than 200 ms when the heart rate is slower than 120 BPM. After the pruning process, only the peaks corresponding to AO and RF are in the candidate set, e.g., peaks 3, 14, 2, 11, 1 and 10 in Figure 7(a). As there are multiple heartbeat cycles in the two-second SCG sequence, there are multiple candidates of AO and RF peaks. We choose the two peaks that are the closest to each other as the AO and RF for the coarse-template since the AO-RF interval is usually smaller than the RF-AO interval. We then measure the interval µ between the selected AO and RF peaks. We use a segment with a duration of 1.5µ as the coarse-template, starting from 0.5µ before the AO peak to include the ATC stage. The resulting coarse-template is shown the blue dashed rectangle in Figure 7(a). In the third step, we perform a cross-correlation between the coarse-template with the continuous SCG signals. Figure 7(b) shows the result of the correlation, where each heartbeat corresponds to a peak that is easier to be identified than the AO or RF peaks in the raw SCG. We use a threshold based scheme to detect peaks in the correlation result. By measuring the number of correlation peaks, we can derive the heartbeat interval and the heart rates of the given SCG signal. The coarse-template based scheme gives more stable heart rate estimation than auto-correlation or FFT based schemes. This is because the similarity of AO and RF peaks for some user may introduce multiple peaks in the auto-correlation and FFT of the SCG signal, which leads to large errors in heart rate estimation. Due to the variations in the timing and amplitude of the AO and RF peaks, our coarse-template could be imprecise. For example, our heuristic algorithm could select a wrong peak to be the AO or RF. Moreover, the coarse-template derived from a single heartbeat cycle could be noisy due to the interference from breathing or other micro movements when collecting the SCG samples. Consequently, the segmentation result based on correlation of the coarse-template is not well aligned. As shown in Figure 7(c), the segmentation results of fifteen heartbeat samples collected at different times from the same person are not perfectly aligned with each other due to the errors in the coarse-template. 4.3 Fine-grained Alignment We use a fine-alignment-template to help align the SCG signals that are collected under different conditions. The fine-alignment-template is produced in the training process and we generate one fine-template for each scenario. For a new SCG sequence, we first use the heart rate estimation scheme to get the heart rates that are used for selecting the fine-template. Note that the fine-template generated in the training process can be applied to all SCG samples with a similar heart rate. Therefore, we only need to generate the fine-template once. We use an SCG sequence that contains at least ten consecutive heartbeats to generate the fine-template. First, we use the coarse segmentation scheme to divide the heartbeat signal into individual cycles. Second, we average over all the heartbeat cycles to reduce the impact of occasionally misaligned cycles and the noises caused by micro-movements. The smoothed signal is shown in Figure 8(a). Third, we use the smoothed signal to estimate the Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018
140:10·L.Wang et al. Averaged signa c0. -Coarse alignment -Fine alignment 0.6 0.4 -0.2 0.2 0.2 -0. 0.4 0 0 02 0.40.60.8 0.20.40.6 100 200 300 Time(s) Time(s) Deviation(ms) (a)Fine-alignment-template selection (b)Fine alignment results (c)CDF of the time deviations in different alignment schemes Fig.8.Fine alignment results start of the ATC stage,instead of using a heuristic interval in the coarse-template.We observe that the smoothed SCG signal remains almost static before the ATC stage and starts to change drastically at the ATC stage.Thus, to estimate the start of the ATC stage,we first normalize the amplitude of the smoothed signal by dividing the samples by the maximum amplitude of the signal.We then estimate the first derivative of the smoothed signal S'(t)=ds(t)/dt using the expression S(t)S(t +m)-S(t),where we take the time difference m as four sample points(i.e.,40 ms at a sampling rate of 100 Hz).As shown in Figure 8(a),the first derivative of the SCG signal, S'(t),has a high amplitude at the start of ATC.Therefore,we use a threshold based scheme to detect the ATC start on the normalized SCG signal.We use the smoothed SCG signal between the ATC starting point and the RF as the fine-alignment-template,see Figure 8(a). The fine-template is used for aligning the heartbeat cycles in a testing continuous heartbeat sequence.We perform a cross-correlation between the fine-template and the testing sequence.Note that the fine-template should have a similar heart rate as the testing sequence,as it is selected based on the heart rate estimation. Therefore,by locating the peaks in the cross-correlation result,we can accurately align the starting point of the ATC stage of different heartbeat cycles.Figure 8(b)shows the aligned of fifteen heartbeat cycles collected over a period of three days for a user.We observe that our fine alignment scheme can precisely match the key features of the AO-RF interval.To evaluate the performance of the alignment scheme,we collected SCG signals from five users,each containing 100 heartbeat cycles.Figure 8(c)shows the CDF of alignment deviations for the heart rate estimation algorithm and the fine alignment algorithm.For the alignment achieved by the coarse-template,the average deviation is 45.23 ms,which is much larger than the average deviation of 9.02 ms from the fine alignment algorithm. 5 FEATURE EXTRACTION In this section,we focus on extracting features for user authentication from the SCG signals.Firstly,we preprocess the SCG signals to normalize both the amplitude and the length of the heartbeat signals.Secondly,we use the wavelet-based method to extract one set of feature vectors from each heartbeat cycle. 5.1 Normalization The normalization algorithm takes the aligned heartbeat signals and uses two steps to reduce the variations of the SCG signals.The first step is to reduce the variation of the SCG amplitude so that heartbeats collected under different conditions have comparable amplitudes.The amplitude of SCG signals depends on the angle between the mobile phone's y-axis and the chest of the user,the position of the mobile phone,and the pressure that the user applied to the phone when collecting the heartbeat signal.Our system allows the user to collect the SCG signals in slightly different ways.Therefore,the amplitudes of the SCG signals collected under different conditions are Proc.ACM Interact.Mob.Wearable Ubiquitous Technol.,Vol.2,No.3,Article 140.Publication date:September 2018
140:10 • L. Wang et al. 0 0.2 0.4 0.6 0.8 Time (s) -0.3 -0.2 -0.1 0 0.1 0.2 Acceleration (m/s 2 ) Averaged signal Derivative Threshold line Fine template (a) Fine-alignment-template selection 0 0.2 0.4 0.6 Time(s) -0.4 -0.2 0 0.2 0.4 Acceleration (m/s 2 ) (b) Fine alignment results 0 100 200 300 Deviation (ms) 0 0.2 0.4 0.6 0.8 1 CDF Coarse alignment Fine alignment (c) CDF of the time deviations in different alignment schemes Fig. 8. Fine alignment results start of the ATC stage, instead of using a heuristic interval in the coarse-template. We observe that the smoothed SCG signal remains almost static before the ATC stage and starts to change drastically at the ATC stage. Thus, to estimate the start of the ATC stage, we first normalize the amplitude of the smoothed signal by dividing the samples by the maximum amplitude of the signal. We then estimate the first derivative of the smoothed signal S ′ (t) = dS (t)/dt using the expression S (t) ≈ S (t +m) − S (t), where we take the time difference m as four sample points (i.e., 40 ms at a sampling rate of 100 Hz). As shown in Figure 8(a), the first derivative of the SCG signal, S ′ (t), has a high amplitude at the start of ATC. Therefore, we use a threshold based scheme to detect the ATC start on the normalized SCG signal. We use the smoothed SCG signal between the ATC starting point and the RF as the fine-alignment-template, see Figure 8(a). The fine-template is used for aligning the heartbeat cycles in a testing continuous heartbeat sequence. We perform a cross-correlation between the fine-template and the testing sequence. Note that the fine-template should have a similar heart rate as the testing sequence, as it is selected based on the heart rate estimation. Therefore, by locating the peaks in the cross-correlation result, we can accurately align the starting point of the ATC stage of different heartbeat cycles. Figure 8(b) shows the aligned of fifteen heartbeat cycles collected over a period of three days for a user. We observe that our fine alignment scheme can precisely match the key features of the AO-RF interval. To evaluate the performance of the alignment scheme, we collected SCG signals from five users, each containing 100 heartbeat cycles. Figure 8(c) shows the CDF of alignment deviations for the heart rate estimation algorithm and the fine alignment algorithm. For the alignment achieved by the coarse-template, the average deviation is 45.23 ms, which is much larger than the average deviation of 9.02 ms from the fine alignment algorithm. 5 FEATURE EXTRACTION In this section, we focus on extracting features for user authentication from the SCG signals. Firstly, we preprocess the SCG signals to normalize both the amplitude and the length of the heartbeat signals. Secondly, we use the wavelet-based method to extract one set of feature vectors from each heartbeat cycle. 5.1 Normalization The normalization algorithm takes the aligned heartbeat signals and uses two steps to reduce the variations of the SCG signals. The first step is to reduce the variation of the SCG amplitude so that heartbeats collected under different conditions have comparable amplitudes. The amplitude of SCG signals depends on the angle between the mobile phone’s y-axis and the chest of the user, the position of the mobile phone, and the pressure that the user applied to the phone when collecting the heartbeat signal. Our system allows the user to collect the SCG signals in slightly different ways. Therefore, the amplitudes of the SCG signals collected under different conditions are Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 2, No. 3, Article 140. Publication date: September 2018