Tremor Detection Using Smartphone-based Acoustic Sensing Wei Wang Xun Wang Abstract State Key Laboratory for Novel State Key Laboratory for Novel In this demo,we present a tremor detection application on Software Technology,Nanjing Software Technology,Nanjing smartphones for early diagnosis of Parkinson's disease. University University Our tremor detection scheme uses inaudible sound emit- Nanjing,210023,china Nanjing,210023,china ted by the built-in speaker on the smartphone.We measure ww@nju.edu.cn 131220134@smail.nju.edu.cn the movement of the hand by extracting the phase of sound waves reflected by the hand.Our application can classify whether the hand is static,moving,or trembling,without re- quiring the patient to hold the phone.The application can Lei Xie also determine the intensity of the tremor by measuring pa- State Key Laboratory for Novel rameters such as the trembling frequency and the trembling Software Technology,Nanjing magnitude. University Nanjing,210023,china Author Keywords Ixie@nju.edu.cn Ultrasound;Tremor Detection;Smartphone;Device-free ACM Classification Keywords H.5.m [Information interfaces and presentation (e.g.,HCI)]: Miscellaneous Paste the appropriate copyright statement here.ACM now supports three ditferent copyright statements: Introduction .ACM copyight:ACM holds the copyright on the work.This is the historical approach. Tremor is an important symptom associated with disorders License:The author(s)retain copyright,but ACM recelves an exclusive such as Parkinson's Disease(PD)or Alzheimer's disease publication license. (AD).Tremor detection and classification are important for Open Access:The author(s)wish to pay for the work to be open access.The additlonal fee must be paid to ACM early diagnosis of these disorders.Therefore,it is important This text field is large enough to hold the appropriate release statement assuming it is single spaced in a sans-serif 7 point tont. to develop a convenient way to monitor and record tremor Every submission will be assigned their own unique DOI string to be included here. behaviors that can be used by anyone at any place
Tremor Detection Using Smartphone-based Acoustic Sensing Wei Wang State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, 210023, china ww@nju.edu.cn Xun Wang State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, 210023, china 131220134@smail.nju.edu.cn Lei Xie State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, 210023, china lxie@nju.edu.cn Paste the appropriate copyright statement here. ACM now supports three different copyright statements: • ACM copyright: ACM holds the copyright on the work. This is the historical approach. • License: The author(s) retain copyright, but ACM receives an exclusive publication license. • Open Access: The author(s) wish to pay for the work to be open access. The additional fee must be paid to ACM. This text field is large enough to hold the appropriate release statement assuming it is single spaced in a sans-serif 7 point font. Every submission will be assigned their own unique DOI string to be included here. Abstract In this demo, we present a tremor detection application on smartphones for early diagnosis of Parkinson’s disease. Our tremor detection scheme uses inaudible sound emitted by the built-in speaker on the smartphone. We measure the movement of the hand by extracting the phase of sound waves reflected by the hand. Our application can classify whether the hand is static, moving, or trembling, without requiring the patient to hold the phone. The application can also determine the intensity of the tremor by measuring parameters such as the trembling frequency and the trembling magnitude. Author Keywords Ultrasound; Tremor Detection; Smartphone; Device-free ACM Classification Keywords H.5.m [Information interfaces and presentation (e.g., HCI)]: Miscellaneous Introduction Tremor is an important symptom associated with disorders such as Parkinson’s Disease (PD) or Alzheimer’s disease (AD). Tremor detection and classification are important for early diagnosis of these disorders. Therefore, it is important to develop a convenient way to monitor and record tremor behaviors that can be used by anyone at any place
Our hand movement tracking system is based on the Low Latency Acoustic Phase(LLAP)technology [5]1.By mea- suring the phase of sound waves reflected by the hand. Tremor Detection LLAP provides millimeter level accuracy in movement dis- tances.In this way,our tremor detection application can Speaker determine whether the hand is static,moving,or trembling. Our application also measures the frequency and magni- tude of the tremor.With these measurements,our appli- cation can classify the tremor to light and severe tremors, Microphone as shown in Figure 2.For example,the PD tremor is a typ- ical resting tremor that has frequencies of 4~6Hz [1].To Start Test Figure 1:Demonstration scenario our best knowledge,this application is the first contact-free tremor detection scheme that can run on commercial smart- History Existing tremor detection schemes either use Laser based phones. [1],electromyography(EMG)based [2],or accelerometer System Design based measurements [3,6].Laser and EMG based tremor Our tremor detection system runs as an iOS application on detection require specialized equipment that is only avail- Figure 2:User interface design commercial iPhones without any hardware modifications. able in hospitals and clinics.Accelerometer based schemes utilize the Inertial Measurement Unit(IMU)in smartphones Our system contains two components:Hand tracking and tremor detection,as shown in Figure 3. or smartwatches to detect hand tremor so that users can perform the test at almost everywhere.However.the pa- Hand Tracking tient needs to hold the phone or wear a smartwatch while The hand tracking component uses sound wave reflected Sound performing the test.The weight and pressure introduced Reflection by the hand to measure hand movements [5].It first sends by the smartphone/smartwatch may significantly impact the human-inaudible sound through the speaker of the smart- Hand testing results.Furthermore,it is difficult to detect intention phone in eight frequencies from 16,800Hz to 21,700Hz, Tracking tremor and task-specific tremor when the user is holding a with a frequency interval of 700Hz.It then uses the built-in smartphone. microphone to record the reflected sound with a sampling dE“rC0 rate of 48kHz.After converting the received sound into the In this demo.we present a tremor detection system that Tremor baseband signal by mixing it with the sending frequencies. Detection uses acoustic signals emitted by smartphones,as illus- LLAP calculates the phase change caused by hand move- trated in Figure 1.Our tremor detection system is device- ments by subtracting the static signals caused by Line-Of- Moving free so that users do not need to hold or wear any devices. Sight path and reflections of static objects,such as walls They just need to put their hand/arm close to the mobile and tables.The key advantage of LLAP is that the phase Figure 3:Tremor detection system phone (within 30cm)to perform the test.Our system works design for various use cases,including the case that the user is 1The LLAP system has been presented and demonstrated in Mobi- holding an object,such as a pen or a spoon. Com201651
Speaker Microphone Figure 1: Demonstration scenario Existing tremor detection schemes either use Laser based [1], electromyography (EMG) based [2], or accelerometer based measurements [3, 6]. Laser and EMG based tremor detection require specialized equipment that is only available in hospitals and clinics. Accelerometer based schemes utilize the Inertial Measurement Unit (IMU) in smartphones or smartwatches to detect hand tremor so that users can perform the test at almost everywhere. However, the patient needs to hold the phone or wear a smartwatch while performing the test. The weight and pressure introduced by the smartphone/smartwatch may significantly impact the testing results. Furthermore, it is difficult to detect intention tremor and task-specific tremor when the user is holding a smartphone. Start Test Normal Light Severe Tremor Detection History Figure 2: User interface design Hand Tracking Sound Reflection Tremor Detection Static Moving Movement distance Tremble Frequency Magnitude Figure 3: Tremor detection system design In this demo, we present a tremor detection system that uses acoustic signals emitted by smartphones, as illustrated in Figure 1. Our tremor detection system is devicefree so that users do not need to hold or wear any devices. They just need to put their hand/arm close to the mobile phone (within 30cm) to perform the test. Our system works for various use cases, including the case that the user is holding an object, such as a pen or a spoon. Our hand movement tracking system is based on the Low Latency Acoustic Phase (LLAP) technology [5] 1 . By measuring the phase of sound waves reflected by the hand, LLAP provides millimeter level accuracy in movement distances. In this way, our tremor detection application can determine whether the hand is static, moving, or trembling. Our application also measures the frequency and magnitude of the tremor. With these measurements, our application can classify the tremor to light and severe tremors, as shown in Figure 2. For example, the PD tremor is a typical resting tremor that has frequencies of 4∼6Hz [1]. To our best knowledge, this application is the first contact-free tremor detection scheme that can run on commercial smartphones. System Design Our tremor detection system runs as an iOS application on commercial iPhones without any hardware modifications. Our system contains two components: Hand tracking and tremor detection, as shown in Figure 3. Hand Tracking The hand tracking component uses sound wave reflected by the hand to measure hand movements [5]. It first sends human-inaudible sound through the speaker of the smartphone in eight frequencies from 16,800Hz to 21,700Hz, with a frequency interval of 700Hz. It then uses the built-in microphone to record the reflected sound with a sampling rate of 48kHz. After converting the received sound into the baseband signal by mixing it with the sending frequencies, LLAP calculates the phase change caused by hand movements by subtracting the static signals caused by Line-OfSight path and reflections of static objects, such as walls and tables. The key advantage of LLAP is that the phase 1The LLAP system has been presented and demonstrated in MobiCom 2016 [5]
Normal Movements Trembling Normal Movements Trembling 0 81012141618 20 2 81012 18 20 Time(second) Time (second) Figure 4:Distance measurement for normal movement and Figure 5:Distance measurement after high-pass filtering trembling estimator for the hand movement trend,which averages the measurements lead to very accurate distance measure- distance measurements over a time period of 200ms.The ments since a distance change equal to the sound wave- second part extracts the tremor signal by subtracting the length of 1.8cm introduces a phase change of 2.There- hand movement trend from the distance measurements. fore,LLAP can track the hand movements with millimeter Figure 5 shows the tremor signal after the high-pass filter accuracy.Furthermore,LLAP provides low latency distance processing for the same sequence shown in Figure 4. information with a small delay of 10.7 milliseconds.In this demo,we use our open source LLAP implementation which Our tremor detection component uses the variation of the is available at [4]. filtered tremor signal and the movement trend to determine whether the hand is static,moving or trembling.Under the Tremor Detection case of a tremor,our system will measure both the trem- We detect hand tremor using the high precision distance bling frequency and the trembling magnitude using the fil- measurements provided by LLAP.Figure 4 shows the dis- tered tremor signal.For example,from Figure 5,the trem- tance output for normal movement and trembling.We ob- bling frequency(which is about 7Hz)can be measured by serve that normal movements have smooth distance changes counting the peaks during a time period of one second.We over time.When the hand starts trembling after 12 sec- can also obtain the trembling magnitude,which is around onds,we can clearly observe the vibrations of the hand. 8mm(peak-to-peak)in this case.The application then de- However,the tremor signals are mixed with hand movement termines the tremor intensity using these measurements. trends,as the user's hand moves around slowly during the trembling period. Implementation and Demo Setup The tremor detection application prototype is implemented We use a high-pass filter to remove the smooth movements using Objective-C on the iOS platform.The application can of the hand and extract the tremor signal.This high-pass be installed on common iPhone models,such as iPhone filter contains two parts.The first part is a moving average 6/6s/7
0 2 4 6 8 10 12 14 16 18 20 Time (second) -60 -40 -20 0 20 Distance (mm) Normal Movements Trembling Figure 4: Distance measurement for normal movement and trembling measurements lead to very accurate distance measurements since a distance change equal to the sound wavelength of 1.8cm introduces a phase change of 2π. Therefore, LLAP can track the hand movements with millimeter accuracy. Furthermore, LLAP provides low latency distance information with a small delay of 10.7 milliseconds. In this demo, we use our open source LLAP implementation which is available at [4]. Tremor Detection We detect hand tremor using the high precision distance measurements provided by LLAP. Figure 4 shows the distance output for normal movement and trembling. We observe that normal movements have smooth distance changes over time. When the hand starts trembling after 12 seconds, we can clearly observe the vibrations of the hand. However, the tremor signals are mixed with hand movement trends, as the user’s hand moves around slowly during the trembling period. We use a high-pass filter to remove the smooth movements of the hand and extract the tremor signal. This high-pass filter contains two parts. The first part is a moving average 0 2 4 6 8 10 12 14 16 18 20 Time (second) -5 0 5 10 Distance (mm) Normal Movements Trembling Figure 5: Distance measurement after high-pass filtering estimator for the hand movement trend, which averages the distance measurements over a time period of 200ms. The second part extracts the tremor signal by subtracting the hand movement trend from the distance measurements. Figure 5 shows the tremor signal after the high-pass filter processing for the same sequence shown in Figure 4. Our tremor detection component uses the variation of the filtered tremor signal and the movement trend to determine whether the hand is static, moving or trembling. Under the case of a tremor, our system will measure both the trembling frequency and the trembling magnitude using the filtered tremor signal. For example, from Figure 5, the trembling frequency (which is about 7Hz) can be measured by counting the peaks during a time period of one second. We can also obtain the trembling magnitude, which is around 8mm (peak-to-peak) in this case. The application then determines the tremor intensity using these measurements. Implementation and Demo Setup The tremor detection application prototype is implemented using Objective-C on the iOS platform. The application can be installed on common iPhone models, such as iPhone 6/6s/7
Our application provides a convenient way to measure and disorders.Furthermore,we plan to use laser or accelerom- Use Cases Method record different types of tremors.The use cases of the ap- eter based systems to calibrate the tremor frequency and Rest tremor Recording tremor plication are listed in Table 1. magnitude measurements and further improve the accuracy monitoring activities multiple of our measurements We will bring our smartphones to demonstrate the tremor times everyday for PD diagnosis. detection application.The demonstration can be setup REFERENCES within 5 minutes. 1.Anne Beuter,Michele S Titcombe,Francois Richer, Daily Performing daily Christian Gross,and Dominique Guehl.2001.Effect of The conference attendees can interact with our demo in the activity activities,such as deep brain stimulation on amplitude and frequency following ways. monitoring eating and writing. characteristics of rest tremor in Parkinson's disease. while putting the Thalamus Related Systems 1,3(2001),203-211. phone on the table. The conference attendees can have hands-on experi- ence on the application,as shown in Figure 1. 2.Carmen Camara,Pedro Isasi,Kevin Warwick,Virginie Ruiz,Tipu Aziz,John Stein,and Eduard Bakstein. Task- Measuring hand- To help the attendees to understand the underlying 2015.Resting tremor classification and detection in specific writing tremor by technology of acoustic sensing,we will also demon- Parkinson's disease patients.Biomedical Signal tremor holding a pen in strate a simple gesture-based game,called"LLAPFly". Processing and Control 16(2015),88-97. the hand. Interacting with this simple game will provide a chance 3.Ivan Garcia-Magarino,Carlos Medrano,Inmaculada for to evaluate the sensitivity and robustness of our Table 1:Use cases for tremor Plaza,and Barbara Olivan.2016.A smartphone-based acoustic sensing scheme. detection. system for detecting hand tremors in unconstrained For interested attendees,we can also show a real- environments.Personal and Ubiguitous Computing 20. time baseband plot on our laptop.This will help the 6(2016),959-971. attendees to understand how the baseband phase is 4.Ke Sun,Wei Wang,and Alex X.Liu.2017.LLAP related to hand movements. opensource platform. https//github.com/Samsonsjarkal/LLAP/.(2017) We will also provide opportunities to install our ap- plications on the attendees'smartphones so that 5.Wei Wang,Alex X.Liu,and Ke Sun.2016.Device-Free they can interact with it on their own devices at any Gesture Tracking Using Acoustic Signals.In time/place. Proceedings of ACM MobiCom. 6.Daryl J Wile,Ranjit Ranawaya,and Zelma HT Kiss. Conclusion and Future Work 2014.Smart watch accelerometry for analysis and In this demo.we have shown the feasibility of device-free diagnosis of tremor.Journal of neuroscience methods tremor detection based on sound signals.In the future,we 230(2014),1-4. will conduct clinical research and develop tremor classifica- tion algorithms that help early diagnosis of different types of
Our application provides a convenient way to measure and record different types of tremors. The use cases of the application are listed in Table 1. We will bring our smartphones to demonstrate the tremor detection application. The demonstration can be setup within 5 minutes. Use Cases Method Rest tremor monitoring Recording tremor activities multiple times everyday for PD diagnosis. Daily activity monitoring Performing daily activities, such as eating and writing, while putting the phone on the table. Taskspecific tremor Measuring handwriting tremor by holding a pen in the hand. Table 1: Use cases for tremor detection. The conference attendees can interact with our demo in the following ways. • The conference attendees can have hands-on experience on the application, as shown in Figure 1. • To help the attendees to understand the underlying technology of acoustic sensing, we will also demonstrate a simple gesture-based game, called “LLAPFly”. Interacting with this simple game will provide a chance for to evaluate the sensitivity and robustness of our acoustic sensing scheme. • For interested attendees, we can also show a realtime baseband plot on our laptop. This will help the attendees to understand how the baseband phase is related to hand movements. • We will also provide opportunities to install our applications on the attendees’ smartphones so that they can interact with it on their own devices at any time/place. Conclusion and Future Work In this demo, we have shown the feasibility of device-free tremor detection based on sound signals. In the future, we will conduct clinical research and develop tremor classification algorithms that help early diagnosis of different types of disorders. Furthermore, we plan to use laser or accelerometer based systems to calibrate the tremor frequency and magnitude measurements and further improve the accuracy of our measurements. REFERENCES 1. Anne Beuter, Michèle S Titcombe, François Richer, Christian Gross, and Dominique Guehl. 2001. Effect of deep brain stimulation on amplitude and frequency characteristics of rest tremor in Parkinson’s disease. Thalamus & Related Systems 1, 3 (2001), 203–211. 2. Carmen Camara, Pedro Isasi, Kevin Warwick, Virginie Ruiz, Tipu Aziz, John Stein, and Eduard Bakštein. 2015. Resting tremor classification and detection in Parkinson’s disease patients. Biomedical Signal Processing and Control 16 (2015), 88–97. 3. Iván García-Magariño, Carlos Medrano, Inmaculada Plaza, and Bárbara Oliván. 2016. A smartphone-based system for detecting hand tremors in unconstrained environments. Personal and Ubiquitous Computing 20, 6 (2016), 959–971. 4. Ke Sun, Wei Wang, and Alex X. Liu. 2017. LLAP opensource platform. https://github.com/Samsonsjarkal/LLAP/. (2017). 5. Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-Free Gesture Tracking Using Acoustic Signals. In Proceedings of ACM MobiCom. 6. Daryl J Wile, Ranjit Ranawaya, and Zelma HT Kiss. 2014. Smart watch accelerometry for analysis and diagnosis of tremor. Journal of neuroscience methods 230 (2014), 1–4