Tracking Human Motions in Photographing: A Context-Aware Energy-Saving Scheme for Smart Phones YAFENG YIN,LEI XIE,YUANYUAN FAN,and SANGLU LU,State Key Laboratory for Novel Software Technology,Nanjing University Due to the portability of smart phones,more and more people tend to take photos with smart phones.How- ever,energy-saving continues to be a thorny problem,since photographing is a rather power hungry function. To extend the battery life of phones while taking photos,we propose a context-aware energy-saving scheme called"SenSave."SenSave senses the user's activities during photographing and adopts suitable energy-saving strategies accordingly.SenSave works based on the observation that a lot of energy during photographing is wasted in preparations before shooting.By leveraging the low power-consuming embedded sensors,such as accelerometer and gyroscope,we can recognize the user's activities and reduce unnecessary energy con- sumption.Besides,by maintaining an activity state machine,SenSave can determine the user's activity pro- gressively and improve the recognition accuracy.Experiment results show that SenSave can recognize the user's activities with an average accuracy of 95.5%and reduce the energy consumption during photographing by 30.0%,when compared to the approach by frequently turning ON/OFF the camera or screen.Additionally, we enhance"SenSave"by introducing an extended Markov chain to predict the next activity state and adopt the energy-saving strategy in advance.Then,we can reduce the energy consumption during photographing 2 by36.1%. CCS Concepts:Human-centered computing-Ubiquitous and mobile computing design and eval- uation methods;Empirical studies in ubiquitous and mobile computing; Additional Key Words and Phrases:Activity sensing.energy saving.smart phones,photographing ACM Reference format: Yafeng Yin,Lei Xie,Yuanyuan Fan,and Sanglu Lu.2017.Tracking Human Motions in Photographing:A Context-Aware Energy-Saving Scheme for Smart Phones.ACM Trans.Sen.Netw.13,4,Article 29(September 2017),37 pages. https:/doi.org/.10.1145/3085578 A preliminary version of this work was presented in Proceedings of the 12th IEEE International Conference on Mobile Ad Hoc and Sensor Systems (IEEE MASS 2015)(Fan et al.2015). This work is supported in part by National Natural Science Foundation of China under Grant Nos.61472185,61373129, 61321491,61502224;JiangSu Natural Science Foundation under Grant No.BK20151390:Fundamental Research Funds for the Central Universities under Grant No.020214380035;2016 Program A for outstanding PhD candidate of Nanjing University under Grant No.201601A008.This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Authors'addresses:Y.Yin,L Xie(corresponding author),Y.Fan,and S.Lu,State Key Laboratory for Novel Software Technology,Nanjing University,Nanjing 210023,China;emails:yyf@dislab.nju.edu.cn,lxie@nju.edu.cn,fyymonica@ dislab.nju.edu.cn,sanglu@nju.edu.cn. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted.To copy otherwise,or republish,to post on servers or to redistribute to lists,requires prior specific permission and/or a fee.Request permissions from permissions@acm.org. ©2017ACM1550-4859/2017/09-ART29$15.00 https:/∥doi.org/10.1145/3085578 ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
29 Tracking Human Motions in Photographing: A Context-Aware Energy-Saving Scheme for Smart Phones YAFENG YIN, LEI XIE, YUANYUAN FAN, and SANGLU LU, State Key Laboratory for Novel Software Technology, Nanjing University Due to the portability of smart phones, more and more people tend to take photos with smart phones. However, energy-saving continues to be a thorny problem, since photographing is a rather power hungry function. To extend the battery life of phones while taking photos, we propose a context-aware energy-saving scheme called “SenSave.” SenSave senses the user’s activities during photographing and adopts suitable energy-saving strategies accordingly. SenSave works based on the observation that a lot of energy during photographing is wasted in preparations before shooting. By leveraging the low power-consuming embedded sensors, such as accelerometer and gyroscope, we can recognize the user’s activities and reduce unnecessary energy consumption. Besides, by maintaining an activity state machine, SenSave can determine the user’s activity progressively and improve the recognition accuracy. Experiment results show that SenSave can recognize the user’s activities with an average accuracy of 95.5% and reduce the energy consumption during photographing by 30.0%, when compared to the approach by frequently turning ON/OFF the camera or screen. Additionally, we enhance “SenSave” by introducing an extended Markov chain to predict the next activity state and adopt the energy-saving strategy in advance. Then, we can reduce the energy consumption during photographing by 36.1%. CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing design and evaluation methods; Empirical studies in ubiquitous and mobile computing; Additional Key Words and Phrases: Activity sensing, energy saving, smart phones, photographing ACM Reference format: Yafeng Yin, Lei Xie, Yuanyuan Fan, and Sanglu Lu. 2017. Tracking Human Motions in Photographing: A Context-Aware Energy-Saving Scheme for Smart Phones. ACM Trans. Sen. Netw. 13, 4, Article 29 (September 2017), 37 pages. https://doi.org/10.1145/3085578 A preliminary version of this work was presented in Proceedings of the 12th IEEE International Conference on Mobile Ad Hoc and Sensor Systems (IEEE MASS 2015) (Fan et al. 2015). This work is supported in part by National Natural Science Foundation of China under Grant Nos. 61472185, 61373129, 61321491, 61502224; JiangSu Natural Science Foundation under Grant No. BK20151390; Fundamental Research Funds for the Central Universities under Grant No. 020214380035; 2016 Program A for outstanding PhD candidate of Nanjing University under Grant No. 201601A008. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Authors’ addresses: Y. Yin, L. Xie (corresponding author), Y. Fan, and S. Lu, State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China; emails: yyf@dislab.nju.edu.cn, lxie@nju.edu.cn, fyymonica@ dislab.nju.edu.cn, sanglu@nju.edu.cn. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2017 ACM 1550-4859/2017/09-ART29 $15.00 https://doi.org/10.1145/3085578 ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:2 Y.Yin et al. 1 INTRODUCTION 1.1 Motivation Nowadays,smart phones are widely used in our daily lives.Due to the portability of smart phones, more and more people tend to take photos with their smart phones,for example,taking photos at a tourist attraction.However,energy-saving continues to be an upsetting problem for smart camera phones,since photographing is a very power-hungry function.For example,according to KS Mobile's(KS Mobile Inc.2014)report in 2014,the application Camera360 Ultimate is listed in first place of the top 10 battery-draining applications for Android.Therefore,the huge energy consumption becomes a non-negligible pain point for the users of smart camera phones.Conse- quently,it is essential to reduce the unnecessary energy consumption during photographing to extend the battery life of smart camera phones. 1.2 Limitations of Prior Art Prior work on energy saving of smart phones can be classified into the following three parts:energy consumption of hardware(Fan et al.2007;Bellosa et al.2003;Rajan et al.2006; Balasubramanian et al.2009),power consumption models,and energy-saving schemes for specific applications.For hardware,Chen et al.(2013a)analyze the power consumption of AMOLED displays in multimedia applications and reveal that camera recording incurs high power cost. LiKamWa et al.(2013)report the experimental and analytical characterization of CMOS image sensors and reveal two energy-proportional mechanisms for energy saving.For models,Dong and Zhong(2011)propose Sesame,with which a mobile system constructs an energy model of itself without any external assistance.Xu et al.(2013)propose a new way called V-edge to generate power models based on battery voltage dynamics.For specific applications,Han et al.(2013)study the energy cost made by human-screen interaction,such as scrolling on the screen.Dietrich and Chakraborty(2013)detect the game's current state and decrease the processor's voltage and fre- quency whenever possible to save energy.Hu et al.(2013)propose a Mobility-Assisted User Contact detection algorithm(MAUC),which utilizes the accelerometer of the phone to detect user move- ments for energy-saving.The Bluetooth scans only when user movements have a high possibility to cause contact changes.LiKamWa et al.(2013)improve the energy efficiency of image sensors based on hardware modifications.There are fewer energy-saving schemes for photographing. Being different from these prior work,we aim to propose an energy-saving scheme for pho- tographing.We aim to recognize the user's activity and reduce unnecessary energy cost when the user is not taking photos.The scheme does not need hardware modifications and user interaction, to guarantee a good user experience. 1.3 Proposed Approach A straight solution to reduce energy cost is to turn off the camera or screen while not taking photos.However,frequently turning ON/OFF the camera or screen is very annoying and leads to a bad user experience.Besides,frequently turning on the camera or screen will incur high energy consumption.Take the Samsung Galaxy Nexus smart phone as an example,the energy consumption of the pair of operations,that is,turning off the camera and screen and then turning on the screen and camera,can keep the camera working on preview mode for about 7s. To propose an efficient energy-saving scheme,we conduct extensive observations.We find that during photographing.a fairly large proportion of energy is wasted in preparations before shoot- ing.For example,the user usually first turns on the camera.Then,he/she will probably adjust the phone time and again,to find a good camera view.Finally,when the camera focuses on the target, the user will press the button to shoot.Between two consecutive shots,the camera works with ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:2 Y. Yin et al. 1 INTRODUCTION 1.1 Motivation Nowadays, smart phones are widely used in our daily lives. Due to the portability of smart phones, more and more people tend to take photos with their smart phones, for example, taking photos at a tourist attraction. However, energy-saving continues to be an upsetting problem for smart camera phones, since photographing is a very power-hungry function. For example, according to KS Mobile’s (KS Mobile Inc. 2014) report in 2014, the application Camera360 Ultimate is listed in first place of the top 10 battery-draining applications for Android. Therefore, the huge energy consumption becomes a non-negligible pain point for the users of smart camera phones. Consequently, it is essential to reduce the unnecessary energy consumption during photographing to extend the battery life of smart camera phones. 1.2 Limitations of Prior Art Prior work on energy saving of smart phones can be classified into the following three parts: energy consumption of hardware (Fan et al. 2007; Bellosa et al. 2003; Rajan et al. 2006; Balasubramanian et al. 2009), power consumption models, and energy-saving schemes for specific applications. For hardware, Chen et al. (2013a) analyze the power consumption of AMOLED displays in multimedia applications and reveal that camera recording incurs high power cost. LiKamWa et al. (2013) report the experimental and analytical characterization of CMOS image sensors and reveal two energy-proportional mechanisms for energy saving. For models, Dong and Zhong (2011) propose Sesame, with which a mobile system constructs an energy model of itself without any external assistance. Xu et al. (2013) propose a new way called V-edge to generate power models based on battery voltage dynamics. For specific applications, Han et al. (2013) study the energy cost made by human-screen interaction, such as scrolling on the screen. Dietrich and Chakraborty (2013) detect the game’s current state and decrease the processor’s voltage and frequency whenever possible to save energy. Hu et al. (2013) propose a Mobility-Assisted User Contact detection algorithm (MAUC), which utilizes the accelerometer of the phone to detect user movements for energy-saving. The Bluetooth scans only when user movements have a high possibility to cause contact changes. LiKamWa et al. (2013) improve the energy efficiency of image sensors based on hardware modifications. There are fewer energy-saving schemes for photographing. Being different from these prior work, we aim to propose an energy-saving scheme for photographing. We aim to recognize the user’s activity and reduce unnecessary energy cost when the user is not taking photos. The scheme does not need hardware modifications and user interaction, to guarantee a good user experience. 1.3 Proposed Approach A straight solution to reduce energy cost is to turn off the camera or screen while not taking photos. However, frequently turning ON/OFF the camera or screen is very annoying and leads to a bad user experience. Besides, frequently turning on the camera or screen will incur high energy consumption. Take the Samsung Galaxy Nexus smart phone as an example, the energy consumption of the pair of operations, that is, turning off the camera and screen and then turning on the screen and camera, can keep the camera working on preview mode for about 7s. To propose an efficient energy-saving scheme, we conduct extensive observations. We find that during photographing, a fairly large proportion of energy is wasted in preparations before shooting. For example, the user usually first turns on the camera. Then, he/she will probably adjust the phone time and again, to find a good camera view. Finally, when the camera focuses on the target, the user will press the button to shoot. Between two consecutive shots, the camera works with ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
Tracking Human Motions in Photographing 29:3 the same settings(e.g.,the same preview size),which result in comparable energy consumption to that of shooting photos.If we can recognize the user's activity and detect the duration between two consecutive shots,then we can decrease the screen brightness,preview size,or the preview frame rate to reduce energy cost. In this article,by leveraging activity sensing,we propose a context-aware energy-saving scheme for smart camera phones.Our idea works based on the observation that most smart phones are equipped with low power-consuming sensors,such as the accelerometer and gyroscope.We can leverage these tiny sensors to recognize the user's activities,such that the corresponding energy- saving strategies (e.g,decreasing the screen brightness,decreasing the frame rate,etc.)can be applied.To reduce the error of activity recognition,we maintain an activity state machine to de- termine the activity state progressively.In addition,we also introduce an extended Markov chain to predict the next activity state,to adopt a suitable energy-saving strategy in advance to fur- ther reduce energy cost.Without user interaction,we can reduce the energy consumption during photographing while guaranteeing a good user experience. 1.4 Technique Challenges and Solutions There are some challenges in activity sensing and designing the energy-saving scheme for taking photos with smart phones. -Activity sensing:The first challenge is how to use the sensor data for activity recognition. To address this challenge,we propose a three-level architecture,which classifies the activi- ties into three levels:body level,arm level,and wrist level.For the sensor data of a potential activity,we first utilize the variance and periodicity of the sensor data to classify the activity into one of the three levels.For activities in the same level,we combine data from differ- ent sensors to distinguish one from another based on the features of activities.To reduce the error of activity recognition,we maintain an activity state machine and determine the user's activity state progressively. -Energy-saving scheme design:The second challenge is how to design an appropriate energy-saving scheme with the recognized activities during photographing.To address this challenge,we propose a context-aware energy-saving scheme SenSave,which adopts suit- able energy-saving strategies based on the user's activities.In body level,SenSave focuses on turning ON/OFF sensors,camera,and screen.In arm level,SenSave focuses on adjust- ing the screen brightness,starting or stopping the camera preview.In wrist level,SenSave focuses on adjusting the preview size,the preview frame rate of the camera.In each level, we will adjust the parameters in an energy-saving strategy for the specific activity. -Trade-off between activity sensing and energy saving:The third challenge is how to make an appropriate trade-off between the accuracy of activity sensing and energy con- sumption.Obviously,more types of sensor data and larger sampling rates contribute to higher accuracy of activity sensing,while resulting in more energy consumption.To ad- dress this challenge,we only leverage the low power-consuming sensors like accelerom- eter and gyroscope for activity recognition.When guaranteeing the recognition accuracy, we choose the sampling rates of sensors as small as possible.For further energy saving,we introduce an extended Markov chain to predict the next activity state and adopt the suitable energy-saving strategy in advance. 1.5 Key Contributions We make the following contributions in this article(a preliminary version of this work appeared in Fan et al.(2015)). ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:3 the same settings (e.g., the same preview size), which result in comparable energy consumption to that of shooting photos. If we can recognize the user’s activity and detect the duration between two consecutive shots, then we can decrease the screen brightness, preview size, or the preview frame rate to reduce energy cost. In this article, by leveraging activity sensing, we propose a context-aware energy-saving scheme for smart camera phones. Our idea works based on the observation that most smart phones are equipped with low power-consuming sensors, such as the accelerometer and gyroscope. We can leverage these tiny sensors to recognize the user’s activities, such that the corresponding energysaving strategies (e.g., decreasing the screen brightness, decreasing the frame rate, etc.) can be applied. To reduce the error of activity recognition, we maintain an activity state machine to determine the activity state progressively. In addition, we also introduce an extended Markov chain to predict the next activity state, to adopt a suitable energy-saving strategy in advance to further reduce energy cost. Without user interaction, we can reduce the energy consumption during photographing while guaranteeing a good user experience. 1.4 Technique Challenges and Solutions There are some challenges in activity sensing and designing the energy-saving scheme for taking photos with smart phones. —Activity sensing: The first challenge is how to use the sensor data for activity recognition. To address this challenge, we propose a three-level architecture, which classifies the activities into three levels: body level, arm level, and wrist level. For the sensor data of a potential activity, we first utilize the variance and periodicity of the sensor data to classify the activity into one of the three levels. For activities in the same level, we combine data from different sensors to distinguish one from another based on the features of activities. To reduce the error of activity recognition, we maintain an activity state machine and determine the user’s activity state progressively. —Energy-saving scheme design: The second challenge is how to design an appropriate energy-saving scheme with the recognized activities during photographing. To address this challenge, we propose a context-aware energy-saving scheme SenSave, which adopts suitable energy-saving strategies based on the user’s activities. In body level, SenSave focuses on turning ON/OFF sensors, camera, and screen. In arm level, SenSave focuses on adjusting the screen brightness, starting or stopping the camera preview. In wrist level, SenSave focuses on adjusting the preview size, the preview frame rate of the camera. In each level, we will adjust the parameters in an energy-saving strategy for the specific activity. —Trade-off between activity sensing and energy saving: The third challenge is how to make an appropriate trade-off between the accuracy of activity sensing and energy consumption. Obviously, more types of sensor data and larger sampling rates contribute to higher accuracy of activity sensing, while resulting in more energy consumption. To address this challenge, we only leverage the low power-consuming sensors like accelerometer and gyroscope for activity recognition. When guaranteeing the recognition accuracy, we choose the sampling rates of sensors as small as possible. For further energy saving, we introduce an extended Markov chain to predict the next activity state and adopt the suitable energy-saving strategy in advance. 1.5 Key Contributions We make the following contributions in this article (a preliminary version of this work appeared in Fan et al. (2015)). ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:4 Y.Yin et al. (a)Walking (b)Lifting up the (c)Rotating the phone(d)Fine-tuning and (e)Laying down arm shooting the arm Fig.1.Human activities during photographing. -First,we propose a context-aware energy-saving scheme for smart camera phones,by lever- aging the built-in sensors for activity sensing.Based on the activity recognition results,we can adopt corresponding energy-saving strategies. -Second,we build a three-level architecture for activity sensing,including body level,arm level,and wrist level.We use the low power-consuming sensors like accelerometer and gyroscope to extract representative features to distinguish one activity from another.By maintaining an activity state machine,we can determine the user's activity progressively and reduce the error of activity recognition. -Third,we design an efficient energy-saving scheme,which can adaptively choose a suitable energy-saving strategy without user interaction,according to the activity state.Besides, we also introduce an extended Markov chain to predict the next activity state,to adopt a suitable energy-saving strategy in advance for further energy saving. -Fourth,we have implemented a system prototype in android-powered smart camera phones. The experiment results show that our solution is able to recognize the user's activities with an average accuracy of 95.5%.Besides,we can reduce the energy consumption during pho- tographing by 30.0%,when compared to the approach by frequently turning ON/OFF the camera or screen.By introducing the extended Markov chain,we can reduce the energy consumption during photographing by 36.1%. 2 OBSERVATIONS ON PHOTOGRAPHING 2.1 Human Activities Related to Photographing During photographing,the users tend to have similar activities,as shown in Figure 1.Before or after the user takes photos,he/she may stay motionless or keep moving,for example,walking. jogging,and so on.While taking photos,the user usually lifts up the arm,rotates the phone, makes fine-tuning,shoots a picture,then lays down the arm.We categorize all the activities into three levels.(1)Body level:motionlessness,body movement.(2)Arm level:lifting up the arm,laying down the arm.(3)Wrist level:rotating the phone,making fine-tuning,shooting a picture.If the user wants to take multiple photos,then he/she may keep the camera working on the preview state, to take the next photo conveniently.However,it is rather energy consuming to keep the camera working.Therefore,many users tend to turn off the camera between two consecutive shots,unless he/she needs to take multiple photos in a short time. 2.2 Energy Consumption Related to Photographing According to Figure 1,before shooting a photo,there will be a preparation time,during which the user needs to move his/her locations,adjusts the position of the phone or makes fine-tuning,to find a good camera view.Obviously,if the user keeps the camera working with large preview size, it will incur much energy consumption,because the preparation time cannot be ignored.Besides, ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:4 Y. Yin et al. Fig. 1. Human activities during photographing. —First, we propose a context-aware energy-saving scheme for smart camera phones, by leveraging the built-in sensors for activity sensing. Based on the activity recognition results, we can adopt corresponding energy-saving strategies. —Second, we build a three-level architecture for activity sensing, including body level, arm level, and wrist level. We use the low power-consuming sensors like accelerometer and gyroscope to extract representative features to distinguish one activity from another. By maintaining an activity state machine, we can determine the user’s activity progressively and reduce the error of activity recognition. —Third, we design an efficient energy-saving scheme, which can adaptively choose a suitable energy-saving strategy without user interaction, according to the activity state. Besides, we also introduce an extended Markov chain to predict the next activity state, to adopt a suitable energy-saving strategy in advance for further energy saving. —Fourth, we have implemented a system prototype in android-powered smart camera phones. The experiment results show that our solution is able to recognize the user’s activities with an average accuracy of 95.5%. Besides, we can reduce the energy consumption during photographing by 30.0%, when compared to the approach by frequently turning ON/OFF the camera or screen. By introducing the extended Markov chain, we can reduce the energy consumption during photographing by 36.1%. 2 OBSERVATIONS ON PHOTOGRAPHING 2.1 Human Activities Related to Photographing During photographing, the users tend to have similar activities, as shown in Figure 1. Before or after the user takes photos, he/she may stay motionless or keep moving, for example, walking, jogging, and so on. While taking photos, the user usually lifts up the arm, rotates the phone, makes fine-tuning, shoots a picture, then lays down the arm. We categorize all the activities into three levels. (1) Body level: motionlessness, body movement. (2) Arm level: lifting up the arm, laying down the arm. (3) Wrist level: rotating the phone, making fine-tuning, shooting a picture. If the user wants to take multiple photos, then he/she may keep the camera working on the preview state, to take the next photo conveniently. However, it is rather energy consuming to keep the camera working. Therefore, many users tend to turn off the camera between two consecutive shots, unless he/she needs to take multiple photos in a short time. 2.2 Energy Consumption Related to Photographing According to Figure 1, before shooting a photo, there will be a preparation time, during which the user needs to move his/her locations, adjusts the position of the phone or makes fine-tuning, to find a good camera view. Obviously, if the user keeps the camera working with large preview size, it will incur much energy consumption, because the preparation time cannot be ignored. Besides, ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
Tracking Human Motions in Photographing 29:5 3200 280 多 70 240 E2000 1600 10 1200 10 60 400 P1 P2 P3 U1 U2 U3 U4 U5 U6 U7 U8 U9 U10 P1 P2 P3 (a)Energy consumption of three (b)Probability of turning ON/OFF(c)Energy consumption of built-in components camera or screen of 10 users sensors Fig.2.In(a)and(c):P1,Samsung Galaxy Nexus;P2,Samsung Galaxy S5;P3,Samsung Galaxy Note4. frequently turning on/off the camera can also incur extra energy consumption.To optimize the energy-consuming parts and propose an efficient energy-saving scheme,we first use Monsoon power monitor(Monsoon Solutions Inc.2015)to measure the energy cost in photographing. 2.2.1 Energy Consumption in Preparation for Photographing.Power consumption in preparation time of photographing is large.We observe the power consumption in the following three android- based phones,that is,Samsung Galaxy Nexus,Samsung Galaxy S5,and Samsung Galaxy Note4. In Figure 2(a),we show the power of the phone for three components,that is,"Base,""Display," and "Camera."Here,"Base"means the power when the screen is turned off and the phone stays in the idle state,that is,no app runs except for the system program."Display"represents the added power(ie.,compared with"Base")by keeping the screen on."Camera"represents the added power (i.e.,compared with"Display"and "Base")by keeping the camera working in preview mode with default settings.During the experiment,we measure the energy consumption in an office with the same light conditions,while each phone adjusts its screen brightness in an automatic way. According to Figure 2(a),the power of keeping the screen on is dozens of times larger than that of "Base,"while the power of keeping the camera working is several times of that of"Display." Therefore,when keeping the camera working in preview mode,much energy will be wasted in preparation for photographing.However,if we can recognize the user's activities and detect the preparation time accurately,we can decrease the screen brightness,preview size,preview frame rate,and so on,to reduce the unnecessary energy cost. 2.2.2 Energy Consumption of Turning ON/OFF the Camera.Frequently turning ON/OFF the cam- era and screen is annoying and wasting energy.Usually,when the user needs to take multiple photos in a period of time,he/she tends to turn ON/OFF the camera or screen frequently for energy sav- ing,instead of keeping the camera working all the time.To know how users adopt the common energy-saving schemes,that is,turning ON/OFF the camera or screen,we invite 10 users to take photos freely in our campus for 20min.For each user,when he/she takes photos,there will be an observer,who will record the user's behavior,that is,the number of times for taking photos,turn- ing ON/OFF camera,turning ON/OFF screen,turning ON/OFF camera and screen.Here,"turning ON/OFF"means a pair of operations,that is,turn off and turn on.For example,turning off the camera and turning it on again means one time for turning ON/OFF camera. In regard to the three energy-saving schemes,"Turn ON/OFF Camera"means the user turns off the camera after photographing and turning on the camera again for retaking photos.In the process,the user does not turn on/off the screen."Turn ON/OFF Screen"means the user turns off the screen after photographing and turning on the screen again for retaking photos.Considering the privacy issues,users usually need to unlock the screen(e.g.,enter password),when they turn ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:5 Fig. 2. In (a) and (c): P1, Samsung Galaxy Nexus; P2, Samsung Galaxy S5; P3, Samsung Galaxy Note4. frequently turning on/off the camera can also incur extra energy consumption. To optimize the energy-consuming parts and propose an efficient energy-saving scheme, we first use Monsoon power monitor (Monsoon Solutions Inc. 2015) to measure the energy cost in photographing. 2.2.1 Energy Consumption in Preparation for Photographing. Power consumption in preparation time of photographing is large. We observe the power consumption in the following three androidbased phones, that is, Samsung Galaxy Nexus, Samsung Galaxy S5, and Samsung Galaxy Note4. In Figure 2(a), we show the power of the phone for three components, that is, “Base,” “Display,” and “Camera.” Here, “Base” means the power when the screen is turned off and the phone stays in the idle state, that is, no app runs except for the system program. “Display” represents the added power (i.e., compared with “Base”) by keeping the screen on. “Camera” represents the added power (i.e., compared with “Display” and “Base”) by keeping the camera working in preview mode with default settings. During the experiment, we measure the energy consumption in an office with the same light conditions, while each phone adjusts its screen brightness in an automatic way. According to Figure 2(a), the power of keeping the screen on is dozens of times larger than that of “Base,” while the power of keeping the camera working is several times of that of “Display.” Therefore, when keeping the camera working in preview mode, much energy will be wasted in preparation for photographing. However, if we can recognize the user’s activities and detect the preparation time accurately, we can decrease the screen brightness, preview size, preview frame rate, and so on, to reduce the unnecessary energy cost. 2.2.2 Energy Consumption of Turning ON/OFF the Camera. Frequently turning ON/OFF the camera and screen is annoying and wasting energy. Usually, when the user needs to take multiple photos in a period of time, he/she tends to turn ON/OFF the camera or screen frequently for energy saving, instead of keeping the camera working all the time. To know how users adopt the common energy-saving schemes, that is, turning ON/OFF the camera or screen, we invite 10 users to take photos freely in our campus for 20min. For each user, when he/she takes photos, there will be an observer, who will record the user’s behavior, that is, the number of times for taking photos, turning ON/OFF camera, turning ON/OFF screen, turning ON/OFF camera and screen. Here, “turning ON/OFF” means a pair of operations, that is, turn off and turn on. For example, turning off the camera and turning it on again means one time for turning ON/OFF camera. In regard to the three energy-saving schemes, “Turn ON/OFF Camera” means the user turns off the camera after photographing and turning on the camera again for retaking photos. In the process, the user does not turn on/off the screen. “Turn ON/OFF Screen” means the user turns off the screen after photographing and turning on the screen again for retaking photos. Considering the privacy issues, users usually need to unlock the screen (e.g., enter password), when they turn ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:6 Y.Yin et al. Table 1.Energy Consumption for Different Operations Operation Turn on Turn off Turn on Turn off Phone camera camera screen screen Preview (uAh) (uAh) (uAh) (uAh) (uAh/s) Nexus 324.61 314.39 301.36 275.89 169.18 S5 456.57 380.40 311.86 267.83 198.42 Note4 763.02 714.58 439.64 255.22 272.01 on the screen.Each user participating in our experiments unlocks the screen after he/she turns on the screen.Thus,unlocking the screen is included in the "Turn ON/OFF Screen."In regard to "Turn ON/OFF Camera and Screen,"it means the user turns off both camera and screen after photographing,and then turns on screen and camera for retaking photos. In Figure 2(b),we show the probability that the user "Turn ON/OFF Camera,""Turn ON/OFF Screen,"and "Turn ON/OFF Camera and Screen"during photographing.On average,the users adopt one of the above energy-saving schemes with 65.8%probability,instead of doing nothing. Here,"Do Nothing"means that the user keeps the camera working in preview mode during pho- tographing.It usually occurs in the case when taking multiple photos in a short time."Do Nothing" leads to large energy consumption,as described in Section 2.2.1.Among the three energy-saving schemes,the users prefer to turn off both camera and screen for further energy consumption. On average,the probability of "Turn ON/OFF Camera and Screen"achieves 32.9%.However,fre- quently user interaction(e.g.,touch the screen,press the button)degrades the user experience and it can also increase the energy consumption(Dietrich and Chakraborty 2013). In Table 1,by using Samsung Galaxy Nexus,Samsung Galaxy S5,and Samsung Galaxy Note4, we show the energy consumption of turning ON/OFF camera or screen one time.In regard to the measurement,take "ON/OFF Screen"as an example,we first keep the screen on and then turn off the screen.The "turning off"operation incurs the power saltation,then we record the energy consumption E1 in the following 20s from the moment of saltation.After that,we record the en- ergy consumption E2 while keeping the screen off for 20s.Then,we use E1-E2 to represent the energy consumption of "Turning OFF Screen."Similarly,we can measure the energy consumption of other operations.We invite 10 volunteers to participate in the experiment.Each volunteer re- peats the operations 50 times,and we average the experiment results.After that,we compare the energy consumption of turning ON/OFF camera or screen with that of keeping camera on.Take Samsung Galaxy Nexus for an example,Equation(1)shows that "Turn ON/OFF Camera,""Turn ON/OFF Screen,"and "Turn ON/OFF Camera and Screen"one time is equivalent to keeping cam- era in preview mode for 3.78s,3.41s,and 7.19s,respectively.Similarly,"Turn ON/OFF Camera and Screen"one time in Samsung S5 and Samsung Note4 is equivalent to keeping camera in preview mode for 7.14s and 7.99s,respectively. 324.61uAh+314.39uAh ON/OFF Camera: =3.78s 169.18Ah/s 301.36uAh+275.89uAh ON/OFF Screen: =3.41s (1) 169.18uAh/s ON/OFF Camera and Screen:3.78s +3.41s 7.19s It indicates that frequently turning ON/OFF the camera or screen manually is indeed inconvenient and rather energy consuming.If we can decrease the energy consumption while not requiring the ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:6 Y. Yin et al. Table 1. Energy Consumption for Different Operations Phone Operation Turn on camera (uAh) Turn off camera (uAh) Turn on screen (uAh) Turn off screen (uAh) Preview (uAh/s) Nexus 324.61 314.39 301.36 275.89 169.18 S5 456.57 380.40 311.86 267.83 198.42 Note4 763.02 714.58 439.64 255.22 272.01 on the screen. Each user participating in our experiments unlocks the screen after he/she turns on the screen. Thus, unlocking the screen is included in the “Turn ON/OFF Screen.” In regard to “Turn ON/OFF Camera and Screen,” it means the user turns off both camera and screen after photographing, and then turns on screen and camera for retaking photos. In Figure 2(b), we show the probability that the user “Turn ON/OFF Camera,” “Turn ON/OFF Screen,” and “Turn ON/OFF Camera and Screen” during photographing. On average, the users adopt one of the above energy-saving schemes with 65.8% probability, instead of doing nothing. Here, “Do Nothing” means that the user keeps the camera working in preview mode during photographing. It usually occurs in the case when taking multiple photos in a short time. “Do Nothing” leads to large energy consumption, as described in Section 2.2.1. Among the three energy-saving schemes, the users prefer to turn off both camera and screen for further energy consumption. On average, the probability of “Turn ON/OFF Camera and Screen” achieves 32.9%. However, frequently user interaction (e.g., touch the screen, press the button) degrades the user experience and it can also increase the energy consumption (Dietrich and Chakraborty 2013). In Table 1, by using Samsung Galaxy Nexus, Samsung Galaxy S5, and Samsung Galaxy Note4, we show the energy consumption of turning ON/OFF camera or screen one time. In regard to the measurement, take “ON/OFF Screen” as an example, we first keep the screen on and then turn off the screen. The “turning off” operation incurs the power saltation, then we record the energy consumption E1 in the following 20s from the moment of saltation. After that, we record the energy consumption E2 while keeping the screen off for 20s. Then, we use E1 − E2 to represent the energy consumption of “Turning OFF Screen.” Similarly, we can measure the energy consumption of other operations. We invite 10 volunteers to participate in the experiment. Each volunteer repeats the operations 50 times, and we average the experiment results. After that, we compare the energy consumption of turning ON/OFF camera or screen with that of keeping camera on. Take Samsung Galaxy Nexus for an example, Equation (1) shows that “Turn ON/OFF Camera,” “Turn ON/OFF Screen,”and “Turn ON/OFF Camera and Screen” one time is equivalent to keeping camera in preview mode for 3.78s, 3.41s, and 7.19s, respectively. Similarly, “Turn ON/OFF Camera and Screen” one time in Samsung S5 and Samsung Note4 is equivalent to keeping camera in preview mode for 7.14s and 7.99s, respectively. ON/OFF Camera: 324.61uAh + 314.39uAh 169.18uAh/s = 3.78s ON/OFF Screen: 301.36uAh + 275.89uAh 169.18uAh/s = 3.41s ON/OFF Camera and Screen: 3.78s + 3.41s = 7.19s (1) It indicates that frequently turning ON/OFF the camera or screen manually is indeed inconvenient and rather energy consuming. If we can decrease the energy consumption while not requiring the ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
Tracking Human Motions in Photographing 29:7 Data Collection Energy Saving (Low power-consuming sensors) (Applying different strategies based on recognized activities/states) Body-level Arm-level Wrist-level Accelerometer energy-saving strategies energy-saving strategies cnergy-saving strategies 热 Gyroscope Activity state Activity state Activity state in body-level in arm-level in wrist-level Sensor data Body level Arm level Wrist level Activity Segmentation Motionlessness Lifting up the arm Rotating the phone】 (Based on pause detection) Fine-tuning Data Body movement Laying down the arm Detecting the start/end of an Shooting activity based on the sensor data with low variances. Activity Recognition (Classifying an activity into one of the three levels,and recognizing it in the corresponding level) Fig.3.System architecture. user to frequently turn ON/OFF the camera or screen,then we can extend the battery life and provide a better user experience. 2.2.3 Energy Consumption of Using Built-in Sensors.It is possible to use low power-consuming built-in sensors for activity sensing and reduce the energy consumption during photographing. Figure 2(c)shows the power consumption of running the built-in sensor in Samsung Nexus,Sam- sung S5,and Samsung Note4.All these sensors work with the same sampling rate,that is,100Hz, which is the maximal sampling rate of sensors in Samsung Nexus.When we measure the power consumption of a sensor,we turn off other sensors.We first record the power consumption with- out running any sensors as Pw1.Then,we record the power consumption by running an sensor as Pw2.For Pwi and Pw2,the phone has the same screen display.Besides,we average the power consumption in 20s when the phone is in a steady state,that is,eliminating the power saltation of starting the sensor.After that,we use Pw2-PwI to obtain the power consumption of the sensor. In Figure 2(c),the sensors include accelerometer,linear accelerometer,gravity sensor,gyroscope, and camera.Based on the definition of SensorEvent in Android APIs(Google Inc.2016a,2016c),lin- ear acceleration(linear-acc for short)and gravity data are generated from accelerometer,that is,we use a low-pass filter to isolate the force of gravity,then we use the acceleration to minus the grav- ity data in each axis to obtain the linear acceleration.Therefore,the energy consumption of linear accelerometer is a little larger than that of gravity sensor.Both of them are larger than that of ac- celerometer.In regard to gyroscope,it usually has larger energy consumption than accelerometer (Park et al.2011).However,when we turn on the camera and keep the camera working in preview mode,the increase of power consumption is much larger than that of other sensors.Therefore,we can utilize the low power-consuming built-in sensors (i.e.,accelerometer and gyroscope)of the phone to detect the user's activities and reduce the energy consumption during photographing.A simple example could be decreasing the screen brightness or turning off the screen,when we find that the user is not taking photos. 3 SYSTEM OVERVIEW Based on the above observations,we consider using the built-in sensors of smart phones to detect the user's activities and reduce the unnecessary energy cost in photographing.The architecture of our system is shown in Figure 3.First,we collect the data from low power-consuming built-in sensors,that is,accelerometer and gyroscope,as shown in the Data Collection module.Second,we ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:7 Fig. 3. System architecture. user to frequently turn ON/OFF the camera or screen, then we can extend the battery life and provide a better user experience. 2.2.3 Energy Consumption of Using Built-in Sensors. It is possible to use low power-consuming built-in sensors for activity sensing and reduce the energy consumption during photographing. Figure 2(c) shows the power consumption of running the built-in sensor in Samsung Nexus, Samsung S5, and Samsung Note4. All these sensors work with the same sampling rate, that is, 100Hz, which is the maximal sampling rate of sensors in Samsung Nexus. When we measure the power consumption of a sensor, we turn off other sensors. We first record the power consumption without running any sensors as Pw1. Then, we record the power consumption by running an sensor as Pw2. For Pw1 and Pw2, the phone has the same screen display. Besides, we average the power consumption in 20s when the phone is in a steady state, that is, eliminating the power saltation of starting the sensor. After that, we use Pw2 − Pw1 to obtain the power consumption of the sensor. In Figure 2(c), the sensors include accelerometer, linear accelerometer, gravity sensor, gyroscope, and camera. Based on the definition of SensorEvent in Android APIs (Google Inc. 2016a, 2016c), linear acceleration (linear-acc for short) and gravity data are generated from accelerometer, that is, we use a low-pass filter to isolate the force of gravity, then we use the acceleration to minus the gravity data in each axis to obtain the linear acceleration. Therefore, the energy consumption of linear accelerometer is a little larger than that of gravity sensor. Both of them are larger than that of accelerometer. In regard to gyroscope, it usually has larger energy consumption than accelerometer (Park et al. 2011). However, when we turn on the camera and keep the camera working in preview mode, the increase of power consumption is much larger than that of other sensors. Therefore, we can utilize the low power-consuming built-in sensors (i.e., accelerometer and gyroscope) of the phone to detect the user’s activities and reduce the energy consumption during photographing. A simple example could be decreasing the screen brightness or turning off the screen, when we find that the user is not taking photos. 3 SYSTEM OVERVIEW Based on the above observations, we consider using the built-in sensors of smart phones to detect the user’s activities and reduce the unnecessary energy cost in photographing. The architecture of our system is shown in Figure 3. First, we collect the data from low power-consuming built-in sensors, that is, accelerometer and gyroscope, as shown in the Data Collection module. Second, we ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:8 Y.Yin et al. extract the activity segment from the sensor data,as shown in the Activity Segmentation module. Third,we classify an activity into one of the three levels:body level,arm level,wrist level.Then, we recognize the activity in the corresponding level,as shown in the Activity Recognition module. Fourth,we adopt an appropriate energy-saving strategy based on the recognized activity,as shown in the Energy Saving module.In the following paragraphs,we will describe how to do activity sensing and adopt energy-saving strategies. 3.1 Activity Sensing Based on Section 2.1,the user's activities are categorized into three levels:body level,arm level, wrist level.In each level,there may be more than one activity.Besides,the user can transfer from one activity to another.To reduce the error of activity recognition,we maintain an activity state machine to describe the transfer relationship of the activities and determine the user's activity progressively.When we get the sensor data of an activity,we first utilize the variance and period- icity of sensor data to classify the activity into one of the three levels.Then,we combine the data from different sensors to recognize the specific activity in each level. -Body level:It includes motionlessness and body movement.Motionlessness can be recog- nized with its low variance of linear acceleration (linear-acc for short)and gyroscope data. In regard to body movement,which can be walking,jogging,and so on,we utilize the peri- odicity of an activity to infer whether the current activity belongs to body movement.That is to say,we do not distinguish walking,jogging,and so on,we aim to recognize an activity as body movement or not. -Arm level:It includes lifting up the arm and laying down the arm.We utilize the rela- tionship between the gravity data and linear acceleration to distinguish the two activities. Besides,we use a voting mechanism to improve the recognition accuracy. -Wrist level:It includes rotating the phone,making fine-tuning,and shooting a photo.We make use of a linear SVM model to distinguish them with the following features:variance, mean,maximum value and minimum value of linear acceleration and gyroscope data in each axis. 3.2 Energy Saving Considering the specific feature of each activity,we propose an adaptive energy-saving scheme called "SenSave"for photographing.Our solution SenSave does not need user interaction(e.g.,turn ON/OFF the camera or screen manually)for energy-saving,it utilizes the low power-consuming sensors to detect the user's activity and adopts a corresponding energy-saving strategy adaptively. For example,when you walk,jog or stay motionless,we can keep the screen off.When you lift your arm up,it is better to turn on the screen and adjust the screen brightness based on the ambient light.When you make fine-tuning to observe the camera view before shooting a photo,it is better to make the camera work in preview mode.In this way,we can make context-aware energy-saving strategies for camera phones. 4 SYSTEM DESIGN In this section,we will show how to use the built-in sensors(i.e.,accelerometer and gyroscope)for activity sensing and design an efficient energy-saving scheme for photographing in smart camera phones. ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:8 Y. Yin et al. extract the activity segment from the sensor data, as shown in the Activity Segmentation module. Third, we classify an activity into one of the three levels: body level, arm level, wrist level. Then, we recognize the activity in the corresponding level, as shown in the Activity Recognition module. Fourth, we adopt an appropriate energy-saving strategy based on the recognized activity, as shown in the Energy Saving module. In the following paragraphs, we will describe how to do activity sensing and adopt energy-saving strategies. 3.1 Activity Sensing Based on Section 2.1, the user’s activities are categorized into three levels: body level, arm level, wrist level. In each level, there may be more than one activity. Besides, the user can transfer from one activity to another. To reduce the error of activity recognition, we maintain an activity state machine to describe the transfer relationship of the activities and determine the user’s activity progressively. When we get the sensor data of an activity, we first utilize the variance and periodicity of sensor data to classify the activity into one of the three levels. Then, we combine the data from different sensors to recognize the specific activity in each level. —Body level: It includes motionlessness and body movement. Motionlessness can be recognized with its low variance of linear acceleration (linear-acc for short) and gyroscope data. In regard to body movement, which can be walking, jogging, and so on, we utilize the periodicity of an activity to infer whether the current activity belongs to body movement. That is to say, we do not distinguish walking, jogging, and so on, we aim to recognize an activity as body movement or not. —Arm level: It includes lifting up the arm and laying down the arm. We utilize the relationship between the gravity data and linear acceleration to distinguish the two activities. Besides, we use a voting mechanism to improve the recognition accuracy. —Wrist level: It includes rotating the phone, making fine-tuning, and shooting a photo. We make use of a linear SVM model to distinguish them with the following features: variance, mean, maximum value and minimum value of linear acceleration and gyroscope data in each axis. 3.2 Energy Saving Considering the specific feature of each activity, we propose an adaptive energy-saving scheme called “SenSave” for photographing. Our solution SenSave does not need user interaction (e.g., turn ON/OFF the camera or screen manually) for energy-saving, it utilizes the low power-consuming sensors to detect the user’s activity and adopts a corresponding energy-saving strategy adaptively. For example, when you walk, jog or stay motionless, we can keep the screen off. When you lift your arm up, it is better to turn on the screen and adjust the screen brightness based on the ambient light. When you make fine-tuning to observe the camera view before shooting a photo, it is better to make the camera work in preview mode. In this way, we can make context-aware energy-saving strategies for camera phones. 4 SYSTEM DESIGN In this section, we will show how to use the built-in sensors (i.e., accelerometer and gyroscope) for activity sensing and design an efficient energy-saving scheme for photographing in smart camera phones. ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
Tracking Human Motions in Photographing 299 y-axis x-axis 3-axis Top -axis x-axis y-ax Top z-axis x-axis (a)Coordinates of sensors (b)Hold horizontally naturally (c)Hold horizontally backward Fig.4.Coordinate system of the phone and the attitude of a phone being held. 10 Raw dae eprocs5 ed data 10 100 200 300 400 500 600 Sampling sequence (50Hz) Fig.5.Data preprocessing. 4.1 Activity Sensing 4.1.1 Raw Data Collection.We collect sequential data from accelerometer and gyroscope.From the acceleration,we use a low-pass filter to isolate the gravity data.Then,we use the acceleration to minus the gravity data in each axis to obtain the linear acceleration(Google Inc.2016a,2016c).In the phone,the sensors use a standard 3-axis coordinate system,as shown in Figure 4(a),which may be different from the earth coordinate system.For example,when we hold the phone as shown in Figure 4(b),the gravity data in x-axis x almost equals to g(ie.,9.8m/s2).When we hold the phone as shown in Figure 4(c),x equals to-g(i.e.,-9.8m/s2).Here,g represents the gravity acceleration, which always points towards the ground.It is noteworthy that the direction of the gravity sensor data is opposite to that of the gravity acceleration,according to the definition of SensorEvent in Android APIs(Google Inc.2016a).In this article,we utilize gravity data for activity recognition, while not transforming gravity data to the gravity acceleration.From sensors,we obtain the raw data of linear acceleration(linear-acc for short),gravity data,and gyroscope data,which will be used for following data processing and activity recognition. 4.1.2 Data Preprocessing.The raw sensor data collected from the built-in sensors is usually noisy.To use the sensor data for activity recognition,the data offset and noises should be removed first.For data offset,it means when we keep the device static,the linear-acc and gyroscope data are not equal to zero.Thus,we remove the data offset in each axis.For noises,we use a smooth function to mitigate the effect of noise.As shown in Figure 5,when the sampling rate of the sensor is 50Hz,the blue line shows the raw sensor data of linear acceleration in x-axis.After removing the data offset,we smooth the sensor data in blue line with a 9-point moving average.The smooth function is like a low-pass filter,which removes the high-frequency noise.The red line in Figure 5 represents the smoothed sensor data.The effect of outliers and noises has been mitigated. Unless otherwise specified,we will use the preprocessed sensor data to do activity segmentation for activity detection and recognition,as described in the following sections. ACM Transactions on Sensor Networks,Vol.13,No.4,Article 29.Publication date:September 2017
Tracking Human Motions in Photographing 29:9 Fig. 4. Coordinate system of the phone and the attitude of a phone being held. Fig. 5. Data preprocessing. 4.1 Activity Sensing 4.1.1 Raw Data Collection. We collect sequential data from accelerometer and gyroscope. From the acceleration, we use a low-pass filter to isolate the gravity data. Then, we use the acceleration to minus the gravity data in each axis to obtain the linear acceleration (Google Inc. 2016a, 2016c). In the phone, the sensors use a standard 3-axis coordinate system, as shown in Figure 4(a), which may be different from the earth coordinate system. For example, when we hold the phone as shown in Figure 4(b), the gravity data in x-axis xv almost equals to д (i.e., 9.8m/s2). When we hold the phone as shown in Figure 4(c), xv equals to −д (i.e., −9.8m/s2). Here, д represents the gravity acceleration, which always points towards the ground. It is noteworthy that the direction of the gravity sensor data is opposite to that of the gravity acceleration, according to the definition of SensorEvent in Android APIs (Google Inc. 2016a). In this article, we utilize gravity data for activity recognition, while not transforming gravity data to the gravity acceleration. From sensors, we obtain the raw data of linear acceleration (linear-acc for short), gravity data, and gyroscope data, which will be used for following data processing and activity recognition. 4.1.2 Data Preprocessing. The raw sensor data collected from the built-in sensors is usually noisy. To use the sensor data for activity recognition, the data offset and noises should be removed first. For data offset, it means when we keep the device static, the linear-acc and gyroscope data are not equal to zero. Thus, we remove the data offset in each axis. For noises, we use a smooth function to mitigate the effect of noise. As shown in Figure 5, when the sampling rate of the sensor is 50Hz, the blue line shows the raw sensor data of linear acceleration in x-axis. After removing the data offset, we smooth the sensor data in blue line with a 9-point moving average. The smooth function is like a low-pass filter, which removes the high-frequency noise. The red line in Figure 5 represents the smoothed sensor data. The effect of outliers and noises has been mitigated. Unless otherwise specified, we will use the preprocessed sensor data to do activity segmentation for activity detection and recognition, as described in the following sections. ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017
29:10 Y.Yin et al. Aetivity D Laying Walking up the arm down the arm Walking 200 400 600 800 000 1200 1400 1600 Sampling sequence (50Hz) Fig.6.Linear acceleration. ”y-aXs C3 C4 Rptating the phone Fine-和ning Shooting 600 650 700 750 80 850 900 950 1000 1050 1100 Sampling sequence (50Hz) Fig.7.Gyroscope data in rectangle C. 4.1.3 Activity Segmentation.To detect the user's activity,it is necessary to segment the data of an activity from the sensor data.In Figure 6,we show the sensor data of linear accelerometer during photographing.For the activities in photographing,there is often a short pause between two different activities.In Figure 6,we mark each short pause with a black rectangle,that is, rectangle A,B,and D.Therefore,the linear-acc can be used to separate the following activities: motionlessness,walking(body movement),lifting up the arm,and laying down the arm. However,there still exist some micro activities like rotating the phone,fine-tuning and shoot- ing,which are too gentle to be distinguished by the linear acceleration,as the rectangle C shown in Figure 6.In this case,we use gyroscope data to assist for activity segmentation.This is because gyroscope is more sensitive to micro activities like rotating,fine-tuning and shooting,when com- pared to the linear accelerometer.In Figure 7,we show the gyroscope data corresponding to the activities in rectangle C of Figure 6,and mark the detected short pauses with black rectangles C1, C2,C3,and C4.Therefore,we can use gyroscope data to detect the short pause between different micro activities and separate rotating,fine-tuning and shooting.For shooting,we can also use the screen touch operation to assist for activity recognition. According to the above observations,we utilize the linear acceleration and gyroscope data to do activity segmentation,as shown in the following steps. -Step 1:We introduce a sliding pause window,and then calculate the variances of linear-acc during the pause window.Suppose there are n sampling data of linear-acc located in the slid- ing pause window,we represent the data in x-axis,y-axis,and z-axis as {xa,xa2,...,xan. yal,ya2,...,yanl,and (zal,za2.....zan),respectively.The corresponding resultant linear- acc is ai,ie [1.n].Here,ai=y).The variances of linear-acc during the pause window is s2.which can be calculated based on Equation(2): 后= n-1 ai-a)2. (2) ACM Transactions on Sensor Networks,Vol 13.No.4,Article 29.Publication date:September 2017
29:10 Y. Yin et al. Fig. 6. Linear acceleration. Fig. 7. Gyroscope data in rectangle C. 4.1.3 Activity Segmentation. To detect the user’s activity, it is necessary to segment the data of an activity from the sensor data. In Figure 6, we show the sensor data of linear accelerometer during photographing. For the activities in photographing, there is often a short pause between two different activities. In Figure 6, we mark each short pause with a black rectangle, that is, rectangle A, B, and D. Therefore, the linear-acc can be used to separate the following activities: motionlessness, walking (body movement), lifting up the arm, and laying down the arm. However, there still exist some micro activities like rotating the phone, fine-tuning and shooting, which are too gentle to be distinguished by the linear acceleration, as the rectangle C shown in Figure 6. In this case, we use gyroscope data to assist for activity segmentation. This is because gyroscope is more sensitive to micro activities like rotating, fine-tuning and shooting, when compared to the linear accelerometer. In Figure 7, we show the gyroscope data corresponding to the activities in rectangle C of Figure 6, and mark the detected short pauses with black rectangles C1, C2, C3, and C4. Therefore, we can use gyroscope data to detect the short pause between different micro activities and separate rotating, fine-tuning and shooting. For shooting, we can also use the screen touch operation to assist for activity recognition. According to the above observations, we utilize the linear acceleration and gyroscope data to do activity segmentation, as shown in the following steps. —Step 1: We introduce a sliding pause window, and then calculate the variances of linear-acc during the pause window. Suppose there are n sampling data of linear-acc located in the sliding pause window, we represent the data in x-axis, y-axis, and z-axis as {xa1, xa2,..., xan }, {ya1,ya2,...,yan }, and {za1, za2,..., zan }, respectively. The corresponding resultant linearacc is ai,i ∈ [1,n]. Here, ai = (x2 ai + y2 ai + z2 ai ). The variances of linear-acc during the pause window is s2 a, which can be calculated based on Equation (2): s2 a = 1 n − 1 n i=1 (ai − a) 2 . (2) ACM Transactions on Sensor Networks, Vol. 13, No. 4, Article 29. Publication date: September 2017.