Meta-Activity Recognition:A Wearable Approach for Logic Cognition-based Activity Sensing Lei Xie,Xu Dong,Wei Wang,and Dawei Huang State Key Laboratory for Novel Software Technology,Nanjing University,China Email:Ixie@nju.edu.cn,dongxu@dislab.nju.edu.cn,ww@nju.edu.cn,huangdw @dislab.nju.edu.cn Abstract-Activity sensing has become a key technology for activity.Therefore,traditional activity sensing schemes [1] many ubiquitous applications,such as exercise monitoring and 3]are either based on the user-dependent recognition,which elder care.Most traditional approaches track the human motions requires to record the training data from the current user and perform activity recognition based on the waveform match- ing schemes in the raw data representation level.In regard to to improve the recognition accuracy,or relying on heavy- the complex activities with relatively large moving range,they training,which requires to collect a large quantity of training usually fail to accurately recognize these activities,due to the samples to build the templates.It is essential to propose inherent variations in human activities.In this paper,we propose a brand-new activity sensing scheme,such that the derived a wearable approach for logic cognition-based activity sensing recognition models can be scalable to any arbitrary human scheme in the logical representation level,by leveraging the meta- activity recognition.Our solution extracts the angle profiles from subjects in a user-independent and light-training approach. the raw inertial measurements,to depict the angle variation In this paper,we propose a wearable approach for logic cog- of limb movement in regard to the consistent body coordinate nition-based activity sensing,by leveraging the meta-activity system.It further extracts the meta-activity profiles to depict the recognition in the logical representation level.We mainly focus sequence of small-range activity units in the complex activity By leveraging the least edit distance-based matching scheme,our on the complex activities from human subjects,as shown in solution is able to accurately perform the activity sensing.Based Fig.1.Our approach is based on the observation that when on the logic cognition-based activity sensing,our solution achieves the human subject is performing an arbitrary activity,he/she lightweight-training recognition,which requires a small quantity is experiencing a very similar sequence of small-range-activity of training samples to build the templates,and user-independent units in the logical aspect,despite of the detailed differences in recognition,which requires no training from the specific user. The experiment results in real settings shows that our meta- the waveforms of the raw inertial measurements.We leverage activity recognition achieves an average accuracy of 92%for the notion meta-activity to denote the small-range-activity user-independent activity sensing units which compose a common activity of human subject. Given an arbitrary activity,our approach first extracts the I.INTRODUCTION angle profiles from the raw measurements to depict the angle Nowadays activity sensing has become a key technology for variation of limb movement in the consistent body coordinate many ubiquitous applications such as exercise monitoring and system.Then,it further extracts the meta-activity profiles to elder care.For example,in the daily exercise monitoring,it depict the sequence of small-range-activity units in the specific is essential to figure out what kinds of exercises the human activity.By leveraging the least edit distance-based matching subjects did everyday.The rising of the wearable devices has scheme,our solution is able to accurately perform the activity provided new opportunities for activity sensing during human sensing.Since a scalable recognition model is derived from motion.The wearable devices such as the smart watches are the meta-activity-based templates in the logical representation usually embedded with inertial sensors like the accelerometers, level,our solution achieves lightweight-training recognition, gyroscopes and magnetometers.They are able to continuously which requires a small quantity of training samples to build the track the human subject's movements and classify them into templates,and user-independent recognition,which requires the corresponding activities by matching the waveforms of in- no training from the specific user. ertial measurements against the templates.However,a number of common activities,e.g.,dumbbell curl and rope skipping, belong to the complex activities.The complex activity refers to an activity which has large range of movement and incurs rotations on multiple joints of the limbs,e.g.,the movement 1:Upright 2:Dumbbell 3:Dumbbell 4:Dumbbell 5:Dumbbell has angle change of more than 45 and involves more than arbell Rov ●urH Flies .随tern长速st Triceps Extensio 2 joints of the limbs.Moreover,it usually has two complex aspects:the widespread variations in activity details and the large movement range.Due to the user-specific characters like the heights,limb lengths and moving behaviors,there Rope Skipping 9:Pin 0:Ba exist obvious deviations in the raw inertial measurements from Swing different human subjects during the process of the complex Fig.1.Example Complex Activities
Meta-Activity Recognition: A Wearable Approach for Logic Cognition-based Activity Sensing Lei Xie, Xu Dong, Wei Wang, and Dawei Huang State Key Laboratory for Novel Software Technology, Nanjing University, China Email: lxie@nju.edu.cn, dongxu@dislab.nju.edu.cn, ww@nju.edu.cn, huangdw@dislab.nju.edu.cn Abstract—Activity sensing has become a key technology for many ubiquitous applications, such as exercise monitoring and elder care. Most traditional approaches track the human motions and perform activity recognition based on the waveform matching schemes in the raw data representation level. In regard to the complex activities with relatively large moving range, they usually fail to accurately recognize these activities, due to the inherent variations in human activities. In this paper, we propose a wearable approach for logic cognition-based activity sensing scheme in the logical representation level, by leveraging the metaactivity recognition. Our solution extracts the angle profiles from the raw inertial measurements, to depict the angle variation of limb movement in regard to the consistent body coordinate system. It further extracts the meta-activity profiles to depict the sequence of small-range activity units in the complex activity. By leveraging the least edit distance-based matching scheme, our solution is able to accurately perform the activity sensing. Based on the logic cognition-based activity sensing, our solution achieves lightweight-training recognition, which requires a small quantity of training samples to build the templates, and user-independent recognition, which requires no training from the specific user. The experiment results in real settings shows that our metaactivity recognition achieves an average accuracy of 92% for user-independent activity sensing. I. INTRODUCTION Nowadays activity sensing has become a key technology for many ubiquitous applications such as exercise monitoring and elder care. For example, in the daily exercise monitoring, it is essential to figure out what kinds of exercises the human subjects did everyday. The rising of the wearable devices has provided new opportunities for activity sensing during human motion. The wearable devices such as the smart watches are usually embedded with inertial sensors like the accelerometers, gyroscopes and magnetometers. They are able to continuously track the human subject’s movements and classify them into the corresponding activities by matching the waveforms of inertial measurements against the templates. However, a number of common activities, e.g., dumbbell curl and rope skipping, belong to the complex activities. The complex activity refers to an activity which has large range of movement and incurs rotations on multiple joints of the limbs, e.g., the movement has angle change of more than 45◦ and involves more than 2 joints of the limbs. Moreover, it usually has two complex aspects: the widespread variations in activity details and the large movement range. Due to the user-specific characters like the heights, limb lengths and moving behaviors, there exist obvious deviations in the raw inertial measurements from different human subjects during the process of the complex activity. Therefore, traditional activity sensing schemes [1]– [3] are either based on the user-dependent recognition, which requires to record the training data from the current user to improve the recognition accuracy, or relying on heavytraining, which requires to collect a large quantity of training samples to build the templates. It is essential to propose a brand-new activity sensing scheme, such that the derived recognition models can be scalable to any arbitrary human subjects in a user-independent and light-training approach. In this paper, we propose a wearable approach for logic cognition-based activity sensing, by leveraging the meta-activity recognition in the logical representation level. We mainly focus on the complex activities from human subjects, as shown in Fig. 1. Our approach is based on the observation that when the human subject is performing an arbitrary activity, he/she is experiencing a very similar sequence of small-range-activity units in the logical aspect, despite of the detailed differences in the waveforms of the raw inertial measurements. We leverage the notion meta-activity to denote the small-range-activity units which compose a common activity of human subject. Given an arbitrary activity, our approach first extracts the angle profiles from the raw measurements to depict the angle variation of limb movement in the consistent body coordinate system. Then, it further extracts the meta-activity profiles to depict the sequence of small-range-activity units in the specific activity. By leveraging the least edit distance-based matching scheme, our solution is able to accurately perform the activity sensing. Since a scalable recognition model is derived from the meta-activity-based templates in the logical representation level, our solution achieves lightweight-training recognition, which requires a small quantity of training samples to build the templates, and user-independent recognition, which requires no training from the specific user. 1:Upright Barbell Row 2:Dumbbell Curl 3:Dumbbell Flies 4:Dumbbell Lateral Raise 6: Rope Skipping 7: Butterfly 8: Cable Crossover 9: Ping-Pong Swing 5:Dumbbell Triceps Extension 10: Badminton Swing Fig. 1. Example Complex Activities
There are two key technical challenges in realizing the consistent approach,regardless of the exact orientation of the activity sensing scheme.The first challenge is to realize the human bodies.3)We have implemented a prototype system to activity sensing in a user-independent approach,such that the evaluate the real performance,the experiment results in real derived recognition model can be extended to recognize the settings shows that our meta-activity recognition achieves an activities of any arbitrary human subjects,regardless of the average accuracy of 92%for user-independent activity sensing. detailed differences and inherent deviations in the activities from different human subjects.To address this challenge,we II.PROBLEM FORMULATION propose to leverage the angle profiles,i.e.,the angles between In this paper,we investigate the wearable approach for the specified limb and the coordinate axes,to depict the limb activity sensing,i.e.,a wearable device is worn by the human movements.The angle profiles are able to capture the angle subject to continuously collect the inertial measurements of variation of the limb movements relative to the human body, human motion,then an activity sensing scheme is required to which tackle the deviation details caused by the user-specific accurately recognize the complex activities of limb movements characters like the height.Moreover,we propose the method of from human subjects.The complex activity refers to the kind "meta-activity recognition"to perform activity sensing in the of activity with a large range of movement,such as sit-ups logical representation level,based on the sequence of meta- and dumbbell lateral raise.Without loss of generality,we activity profiles,so as to tackle the variations in the long se- leverage the smart watch to sense the human motions,which quence of small-range activities.Specifically,according to the is embedded with inertial sensors including the accelerometer, inertial measurements collected from human motion,instead of gyroscope and magnetometer. performing the waveform-based matching like dynamic time In this paper,we aim to design an activity sensing scheme, warping,we decompose the complex activity into a sequence by considering the following metrics in system performance: of meta-activities,and use this sequence to recognize the 1)Accuracy:The expected accuracy for the activity sensing complex activity via the least edit distance-based matching. scheme to successfully match a specific activity to a correct The second challenge is to build a consistent scheme to activity should be greater than a specified threshold,e.g.,85%. depict the human motion according to the inertial measure- 2)Time-efficiency:The time delay of the activity recognition ments from the wearable devices.Since the human subjects process should be less than a specified threshold,e.g.,500ms. may perform the activities towards any arbitrary direction 3)User-independence:When performing activity sensing,no during the human motion,this causes the templates for activity training data should be required from the specified user. recognition to depend heavily on the actual direction the 4)Lightweight-training:The essential quantity of the training human body is facing,and further enhances the complexities samples to build the templates should be small enough.. in performing activity sensing due to the inconsistency.To III.MODELING THE HUMAN MOTION address this challenge,we depict all the inertial measurements of human motion in terms of a body coordinate system in A.Coordinate System Transformation a consistent approach.Specifically,according to the gravity In regard to activity sensing,as the raw inertial measure- direction and the magnetic direction extracted from the inertial ments are collected from the embedded inertial sensors in measurements.we transform the measurements from the watch the smart watch,they are measured by reference to the body coordinate system (WCS)to the global coordinate system frame of the smart watch.However,the watch coordinate (GCS).Then,by specifying two signal gestures,i.e.,extending system is continuously changing with the arm/wrist movement the arm to the front and dropping the arm downward,we can during the process of human motion,thus the measurements figure out the orientation of the human body in the global from the watch coordinate system cannot be used as a stable coordinate system according to the measurements in the signal reference for the specified activities.In fact,since the human gestures,thus we further transform the measurements to the subject may be performing the activity towards any arbitrary body coordinate system (BCS). direction,the movements should be depicted as the movement To the best of our knowledge,this paper presents the of arms or legs relative to the human body,regardless of the first study of using the method "meta-activity recognition" absolute moving direction of the limbs.Therefore,in order to for logical cognition-based activity sensing.Specifically,we perform activity sensing in a scalable approach,it is essential make three key contributions in this paper.1)Instead of to transform the measurement of limb movements from the performing waveform matching on the inertial measurements watch coordinate system to the body coordinate system. in the raw data level,we extract the angle profiles to depict 1)From Watch Coordinate System to Global Coordinate the angle variation of limb movements,and leverage the System:Fig.2(a)shows the three axes of the watch coordinate meta-activity profiles to depict the complex activities in the system.The X-axis refers to the direction of the lower arm logical representation level,such that the derived recognition when the watch is worn on the wrist,the Yio-axis refers to the model is scalable enough for the activity recognition on any direction of the strap of the watch,and the 2-axis refers to arbitrary human subjects.2)We build a coordinate system the direction which is perpendicular to the watch surface. transformation scheme to transform the inertial measurement According to the acceleration measurements from the ac- from the watch coordinate system to the body coordinate celerometer,we can extract a constant gravitational accel- system,such that the limb movement can be depicted in a eration as a vector g from the low pass filter (such as
There are two key technical challenges in realizing the activity sensing scheme. The first challenge is to realize the activity sensing in a user-independent approach, such that the derived recognition model can be extended to recognize the activities of any arbitrary human subjects, regardless of the detailed differences and inherent deviations in the activities from different human subjects. To address this challenge, we propose to leverage the angle profiles, i.e., the angles between the specified limb and the coordinate axes, to depict the limb movements. The angle profiles are able to capture the angle variation of the limb movements relative to the human body, which tackle the deviation details caused by the user-specific characters like the height. Moreover, we propose the method of “meta-activity recognition” to perform activity sensing in the logical representation level, based on the sequence of metaactivity profiles, so as to tackle the variations in the long sequence of small-range activities. Specifically, according to the inertial measurements collected from human motion, instead of performing the waveform-based matching like dynamic time warping, we decompose the complex activity into a sequence of meta-activities, and use this sequence to recognize the complex activity via the least edit distance-based matching. The second challenge is to build a consistent scheme to depict the human motion according to the inertial measurements from the wearable devices. Since the human subjects may perform the activities towards any arbitrary direction during the human motion, this causes the templates for activity recognition to depend heavily on the actual direction the human body is facing, and further enhances the complexities in performing activity sensing due to the inconsistency. To address this challenge, we depict all the inertial measurements of human motion in terms of a body coordinate system in a consistent approach. Specifically, according to the gravity direction and the magnetic direction extracted from the inertial measurements, we transform the measurements from the watch coordinate system (WCS) to the global coordinate system (GCS). Then, by specifying two signal gestures, i.e., extending the arm to the front and dropping the arm downward, we can figure out the orientation of the human body in the global coordinate system according to the measurements in the signal gestures, thus we further transform the measurements to the body coordinate system (BCS). To the best of our knowledge, this paper presents the first study of using the method “meta-activity recognition” for logical cognition-based activity sensing. Specifically, we make three key contributions in this paper. 1) Instead of performing waveform matching on the inertial measurements in the raw data level, we extract the angle profiles to depict the angle variation of limb movements, and leverage the meta-activity profiles to depict the complex activities in the logical representation level, such that the derived recognition model is scalable enough for the activity recognition on any arbitrary human subjects. 2) We build a coordinate system transformation scheme to transform the inertial measurement from the watch coordinate system to the body coordinate system, such that the limb movement can be depicted in a consistent approach, regardless of the exact orientation of the human bodies. 3) We have implemented a prototype system to evaluate the real performance, the experiment results in real settings shows that our meta-activity recognition achieves an average accuracy of 92% for user-independent activity sensing. II. PROBLEM FORMULATION In this paper, we investigate the wearable approach for activity sensing, i.e., a wearable device is worn by the human subject to continuously collect the inertial measurements of human motion, then an activity sensing scheme is required to accurately recognize the complex activities of limb movements from human subjects. The complex activity refers to the kind of activity with a large range of movement, such as sit-ups and dumbbell lateral raise. Without loss of generality, we leverage the smart watch to sense the human motions, which is embedded with inertial sensors including the accelerometer, gyroscope and magnetometer. In this paper, we aim to design an activity sensing scheme, by considering the following metrics in system performance: 1) Accuracy: The expected accuracy for the activity sensing scheme to successfully match a specific activity to a correct activity should be greater than a specified threshold, e.g., 85%. 2) Time-efficiency: The time delay of the activity recognition process should be less than a specified threshold, e.g., 500ms. 3) User-independence: When performing activity sensing, no training data should be required from the specified user. 4) Lightweight-training: The essential quantity of the training samples to build the templates should be small enough. . III. MODELING THE HUMAN MOTION A. Coordinate System Transformation In regard to activity sensing, as the raw inertial measurements are collected from the embedded inertial sensors in the smart watch, they are measured by reference to the body frame of the smart watch. However, the watch coordinate system is continuously changing with the arm/wrist movement during the process of human motion, thus the measurements from the watch coordinate system cannot be used as a stable reference for the specified activities. In fact, since the human subject may be performing the activity towards any arbitrary direction, the movements should be depicted as the movement of arms or legs relative to the human body, regardless of the absolute moving direction of the limbs. Therefore, in order to perform activity sensing in a scalable approach, it is essential to transform the measurement of limb movements from the watch coordinate system to the body coordinate system. 1) From Watch Coordinate System to Global Coordinate System: Fig. 2(a) shows the three axes of the watch coordinate system. The Xw-axis refers to the direction of the lower arm when the watch is worn on the wrist, the Yw-axis refers to the direction of the strap of the watch, and the Zw-axis refers to the direction which is perpendicular to the watch surface. According to the acceleration measurements from the accelerometer, we can extract a constant gravitational acceleration as a vector g from the low pass filter (such as
Za directions.Therefore,it is essential to build a body coordinate system (BCS)to depict the limb movements in a consistent approach by reference to the human body. In regard to the body coordinate system.we set the vector corresponding to the heading direction of the human subject to represent the 2o axis.For the horizontal plane which is Yw orthogonal to the Z axis,we set the vector which is parallel Yg to the physical plane of the body to represent the X axis,and (a)The relationships between WCS (b)The relationships between GCS set the vector which is perpendicular to the physical plane of and GCS and BCS the body to represent the Yo axis.Fig.2(b)shows the three axes Fig.2.The relationship between different coordinate systems (X,Yo,Zo)of BCS and the three axes (Xg:Ya Zg)of GCS the Butterworth filter [4])in the watch coordinate system. in regard to the physical plane of the human body,respectively. Moreover,according to the magnetic measurements from the Considering that the human subject can perform the activity magnetometer,we can extract the magnetic force as a vector with different orientations of the physical plane of the body. m in the watch coordinate system.Then,we can build a e.g.,standing on the floor or lying on the floor,in all situations, global coordinate system (GCS)based on the gravity direction we can transform any inertial measurement from the GCS to and magnetic direction in the watch coordinate system.The BCS by also using the direction cosine representation.The procedure is as follows:After we obtain the gravity vector orientation of the body coordinate system relative to the global g,we derive its opposite value and normalize this vector as coordinate system is specified by a 3 x 3 rotation matrix C' Zg=g we then set this vector zg to represent the global in which each column is a unit vector along one of the global Zgaxis as it is in the opposite direction of the gravitational coordinate axes specified in terms of the body coordinate axes. acceleration and it is perpendicular to the horizontal plane. A vector quantity vo defined in GCS is equivalent to the vector After computing the cross product y =g x m,we obtain v=C'.v defined in BCS.In this way,we can transform any a vector y that is perpendicular to the plane determined by inertial measurement from the GCS to the BCS.In Section IV. the two distinct but intersecting lines corresponding to g we will introduce the approach to compute the rotation matrix and m.We normalize this vector as ya= 前·Since the C',by leveraging two signal gestures. vector y is on the horizontal plane,we set this vector ya In regard to the activities where the physical plane of the to represent the global Ya-axis.After that,by computing the human body is continuously changing,e.g.,sit-ups,we can set cross product x=gxy,we obtain a vector x that is orthogonal the initial physical plane of the human body as the reference to the plane determined by the two distinct but intersecting body coordinate system.In this way,each of the following lines corresponding to g and y.We normalize this vector as inertial measurements are measured in terms of the reference xg=to represent the global Xg-axis..Fig.2(a)further body coordinate system shows the relationship between the three axes(x,y and z) of WCS and the three axes (xgy and zg)of GCS. B.Modeling the Human Motion with Meta-Activity To quantify the orientation difference between the watch Each complex activity,e.g.,dumbbell side raise and bent- coordinates and global coordinates,we use the direction cosine over dumbbell laterals,is performed with a large range of representation [5].In the direction cosine representation,the movement.So it can be decomposed into a series of small- orientation of the global coordinate relative to the watch range movements which are sequentially performed over time. coordinate system is specified by a 3 x 3 rotation matrix C, Therefore,we leverage the term meta-activities to denote these in which each column is a unit vector along one of the watch small-range movements.Each meta-activity is defined as a unit coordinate axes specified in terms of the global coordinate movement with logically the minimal granularity in regard to axes.A vector quantity ve defined in the watch coordinate the moving range.We can define the whole set of complex system is equivalent to the vector v=C.v defined in the activities as a set C.and the whole set of meta-activities as a global coordinate system.In this way,we are able to transform set M.Then,according to the above definition,each complex any inertial measurement v from WCS to the corresponding activity cC can be depicted as a series of meta-activities, inertial measurement vg in GCS.During the human motion,i.e.,ci=(mj,...,mj),where mjE M. the directions of g and m are continuously updated in WCS 1)Angle Profiles:In regard to the activity sensing,due to track the three axes of GCS,so as to further update the to the differences in human-specific characters such as the rotation matrix C in a real-time approach. height,arm length,and moving behavior,different human 2)From Global Coordinate System to Body Coordinate subjects may perform the same activity with different speeds System:During the human motion,the human subject may and amplitudes.This causes nonnegligible deviations among be facing any arbitrary direction in regard to the global the inertial measurements of the same activities in both time coordinate system.Hence,although we can derive the inertial domain and space domain.Therefore,the meta-activity should measurement of limb movements in GCS,these measurements be depicted in a scalable approach,such that the activity may not be consistent with each other even if they belong sensing scheme can be tolerant to the variances in the limb to the same activity,due to the differences in the facing movements.However,traditional inertial measurements such
Xw Yw Zw Zg Xg Yg (a) The relationships between WCS and GCS Zg(Zb) Yb Xb G Yg Xg ! ! Watch (b) The relationships between GCS and BCS Fig. 2. The relationship between different coordinate systems the Butterworth filter [4]) in the watch coordinate system. Moreover, according to the magnetic measurements from the magnetometer, we can extract the magnetic force as a vector m in the watch coordinate system. Then, we can build a global coordinate system (GCS) based on the gravity direction and magnetic direction in the watch coordinate system. The procedure is as follows: After we obtain the gravity vector g, we derive its opposite value and normalize this vector as zg = −g kgk , we then set this vector zg to represent the global Zg-axis as it is in the opposite direction of the gravitational acceleration and it is perpendicular to the horizontal plane. After computing the cross product y = g × m, we obtain a vector y that is perpendicular to the plane determined by the two distinct but intersecting lines corresponding to g and m. We normalize this vector as yg = y kyk . Since the vector yg is on the horizontal plane, we set this vector yg to represent the global Yg-axis. After that, by computing the cross product x = g×y, we obtain a vector x that is orthogonal to the plane determined by the two distinct but intersecting lines corresponding to g and y. We normalize this vector as xg = x kxk to represent the global Xg-axis.. Fig. 2(a) further shows the relationship between the three axes (xw, yw and zw) of WCS and the three axes (xg, yg and zg) of GCS. To quantify the orientation difference between the watch coordinates and global coordinates, we use the direction cosine representation [5]. In the direction cosine representation, the orientation of the global coordinate relative to the watch coordinate system is specified by a 3 × 3 rotation matrix C, in which each column is a unit vector along one of the watch coordinate axes specified in terms of the global coordinate axes. A vector quantity vw defined in the watch coordinate system is equivalent to the vector vg = C · vw defined in the global coordinate system. In this way, we are able to transform any inertial measurement vw from WCS to the corresponding inertial measurement vg in GCS. During the human motion, the directions of g and m are continuously updated in WCS to track the three axes of GCS, so as to further update the rotation matrix C in a real-time approach. 2) From Global Coordinate System to Body Coordinate System: During the human motion, the human subject may be facing any arbitrary direction in regard to the global coordinate system. Hence, although we can derive the inertial measurement of limb movements in GCS, these measurements may not be consistent with each other even if they belong to the same activity, due to the differences in the facing directions. Therefore, it is essential to build a body coordinate system (BCS) to depict the limb movements in a consistent approach by reference to the human body. In regard to the body coordinate system, we set the vector corresponding to the heading direction of the human subject to represent the Zb axis. For the horizontal plane which is orthogonal to the Zb axis, we set the vector which is parallel to the physical plane of the body to represent the Xb axis, and set the vector which is perpendicular to the physical plane of the body to represent the Yb axis. Fig. 2(b) shows the three axes (Xb, Yb, Zb) of BCS and the three axes (Xg, Yg, Zg) of GCS in regard to the physical plane of the human body, respectively. Considering that the human subject can perform the activity with different orientations of the physical plane of the body, e.g., standing on the floor or lying on the floor, in all situations, we can transform any inertial measurement from the GCS to BCS by also using the direction cosine representation. The orientation of the body coordinate system relative to the global coordinate system is specified by a 3 × 3 rotation matrix C’, in which each column is a unit vector along one of the global coordinate axes specified in terms of the body coordinate axes. A vector quantity vg defined in GCS is equivalent to the vector vb = C’·vg defined in BCS. In this way, we can transform any inertial measurement from the GCS to the BCS. In Section IV, we will introduce the approach to compute the rotation matrix C’, by leveraging two signal gestures. In regard to the activities where the physical plane of the human body is continuously changing, e.g., sit-ups, we can set the initial physical plane of the human body as the reference body coordinate system. In this way, each of the following inertial measurements are measured in terms of the reference body coordinate system. B. Modeling the Human Motion with Meta-Activity Each complex activity, e.g., dumbbell side raise and bentover dumbbell laterals, is performed with a large range of movement. So it can be decomposed into a series of smallrange movements which are sequentially performed over time. Therefore, we leverage the term meta-activities to denote these small-range movements. Each meta-activity is defined as a unit movement with logically the minimal granularity in regard to the moving range. We can define the whole set of complex activities as a set C, and the whole set of meta-activities as a set M. Then, according to the above definition, each complex activity ci ∈ C can be depicted as a series of meta-activities, i.e., ci = hmj1 , · · · , mjk i, where mj ∈ M. 1) Angle Profiles: In regard to the activity sensing, due to the differences in human-specific characters such as the height, arm length, and moving behavior, different human subjects may perform the same activity with different speeds and amplitudes. This causes nonnegligible deviations among the inertial measurements of the same activities in both time domain and space domain. Therefore, the meta-activity should be depicted in a scalable approach, such that the activity sensing scheme can be tolerant to the variances in the limb movements. However, traditional inertial measurements such
as the linear accelerations are very sensitive to the speeds obvious variances in the acceleration measurements,whereas and amplitudes of the limb movements,which fail to depict the variances in the angle profiles are relatively small.We the meta-activity in a scalable approach.Fortunately,it is further compute the DTW distances between each pair of found that,during the process of limb movements,the angle measurements from different human subjects,and obtain the variations between the limb and the body are much more average distance as the metric to quantify the corresponding stable than the traditional inertial measurements,which are variances.Fig.4(c)shows the DTW distances,respectively,for regardless of the human-specific characters such as the height the activity Dumbbell Curl and Sit-Up.It is found that for both and arm length.Therefore,in this paper,we propose to cases the angle profiles achieve much smaller distances than leverage the angle profiles,i.e.,the angles between the lower the acceleration measurement,which implies that the angle arm and the three axes in the body coordinate system,to depict profile is a more stable metric to depict the human motion. the meta-activities of the limb movements.Specifically,since Body the direction of X axis in the watch coordinate system is zCoordinate consistent with the lower arm direction,we can use the vector System X x to depict the lower arm direction in the body coordinate a●● system'.Fig.3(a)shows the vector x to depict the lower arm Lower Arm direction in the BCS.As shown in Fig.3(b),we respectively Direction denote the angle profiles,i.e.,the angles between the lower arm and the X,Y and Z axes in the BCS,as a,B and In order to compute the angle profiles,we take the angle a as an example,suppose the lower arm vector and the vector X axis of the X-axis are v(v=x)and u,respectively,in the (a)The lower arm direction (b)The angle profiles BCS.Then a can be computed according to the cosine value Fig.3.Derive the angle profiles in the body coordinate system as follows: 2)Meta-Activity Profiles:Ideally,in order to depict the V·u Uruz Vyuy Vzuz limb movements of human subject,the angle profiles of all cos a (1) skeletons in the body coordinate system are required to be √喔+哈+吃V+吃+喔 captured.Nevertheless,since the lower arm usually experi- For any specified value of cos a,there exist two solutions of ences a movement with fairly large range during the process a in the range between 0 and 360.Hence,we first compute of human motion,it is already representative to perform the the corresponding solution a within the range [0,180],we activity sensing based on the angle profiles of the lower arm. For the angle profiles a.B and y,according to the definition. then further determine the value of o as follows: they have the following relationship: ifvy≥0 a= 360°-a if vy<0. (2) (cosa)2+(cosB)2+(cosy)2=1 (5) Similarly,we can compute cos B and cosy accordingly,then Given any two values of the a,B and y,the other one can the values of B and y can be determined as follows: be computed according to Eg.(5).However,it still has two candidate solutions according to the corresponding cosine ifv2≥0 (3) value.Therefore,the arm-direction in the BCS can be uniquely 360°-ifv2<0. determined via the three parameters (o,B,) ifvz≥0 For any specified meta-activity,while it is being performed, 360°-ifvx<0. (4) the angle profiles (a,B,are continuously changing.The meta-activity should have the following properties for any of In this way,the angle profiles (o,B,in the BCS can be the parameters (o,B,):1)The variation range of any angle determined within the range[0°，36o]. profile should be less than a threshold 6,e.g.,30.2)The We further conducted empirical studies to validate the above variation trend of any angle profile should be monotonic,e.g., judgement.We invite four human subjects (a,b,c and d) monotonically increasing or decreasing.3)The time duration with different heights and genders to perform the specified of the meta-activity should be less than a threshold t,e.g.. complex activities,and respectively record the corresponding 500ms.The first property implies that the moving range of acceleration measurements and the angle profiles in regard to the meta-activity should be small enough,the second property each of the axes in the body coordinate system.We normalize implies that the moving direction of the meta-activity should all the measurements to the range [0,1]for fair comparison. be monotonic,the third property implies that the time duration Fig.4(a)and Fig.4(b)respectively shows the acceleration mea- of the meta-activity should be limited,even if the moving surements and angle profiles of the activity Dumbbell Curl. range is still small enough. It is found that,among different human subjects,there exist Therefore,for each dimension of the angle profiles (o,B,Y), As mentioned in Section III,the vector x in BCS can be computed we can uniformly divide the rotation range [0,360]into according to the direction cosine representation,it can be continuously multiple sectors,while the angle of each sector is no greater updated in a real time approach. than the threshold 6.In this way,we can use the specified
as the linear accelerations are very sensitive to the speeds and amplitudes of the limb movements, which fail to depict the meta-activity in a scalable approach. Fortunately, it is found that, during the process of limb movements, the angle variations between the limb and the body are much more stable than the traditional inertial measurements, which are regardless of the human-specific characters such as the height and arm length. Therefore, in this paper, we propose to leverage the angle profiles, i.e., the angles between the lower arm and the three axes in the body coordinate system, to depict the meta-activities of the limb movements. Specifically, since the direction of Xw axis in the watch coordinate system is consistent with the lower arm direction, we can use the vector xw to depict the lower arm direction in the body coordinate system1 . Fig. 3(a) shows the vector xw to depict the lower arm direction in the BCS. As shown in Fig. 3(b), we respectively denote the angle profiles, i.e., the angles between the lower arm and the X, Y and Z axes in the BCS, as α, β and γ. In order to compute the angle profiles, we take the angle α as an example, suppose the lower arm vector and the vector of the X-axis are v (v = xw ) and u, respectively, in the BCS. Then α can be computed according to the cosine value as follows: cos α = v · u |v||u| = q vxux + vyuy + vzuz v 2 x + v 2 y + v 2 z q u 2 x + u 2 y + u 2 z . (1) For any specified value of cos α, there exist two solutions of α in the range between 0 ◦ and 360◦ . Hence, we first compute the corresponding solution αb within the range [0◦ , 180◦ ], we then further determine the value of α as follows: α = αb if vy ≥ 0 360◦ − αb if vy < 0. (2) Similarly, we can compute cos β and cos γ accordingly, then the values of β and γ can be determined as follows: β = ( βb if vz ≥ 0 360◦ − βb if vz < 0. (3) γ = γb if vx ≥ 0 360◦ − γb if vx < 0. (4) In this way, the angle profiles hα, β, γi in the BCS can be determined within the range [0◦ , 360◦ ]. We further conducted empirical studies to validate the above judgement. We invite four human subjects (a, b, c and d) with different heights and genders to perform the specified complex activities, and respectively record the corresponding acceleration measurements and the angle profiles in regard to each of the axes in the body coordinate system. We normalize all the measurements to the range [0, 1] for fair comparison. Fig.4(a) and Fig.4(b) respectively shows the acceleration measurements and angle profiles of the activity Dumbbell Curl. It is found that, among different human subjects, there exist 1As mentioned in Section III, the vector xw in BCS can be computed according to the direction cosine representation, it can be continuously updated in a real time approach. obvious variances in the acceleration measurements, whereas the variances in the angle profiles are relatively small. We further compute the DTW distances between each pair of measurements from different human subjects, and obtain the average distance as the metric to quantify the corresponding variances. Fig.4(c) shows the DTW distances, respectively, for the activity Dumbbell Curl and Sit-Up. It is found that for both cases the angle profiles achieve much smaller distances than the acceleration measurement, which implies that the angle profile is a more stable metric to depict the human motion. Z X Y Body Coordinate System Lower Arm Direction xw (a) The lower arm direction Z axis X axis Y axis xw α β γ (b) The angle profiles Fig. 3. Derive the angle profiles in the body coordinate system 2) Meta-Activity Profiles: Ideally, in order to depict the limb movements of human subject, the angle profiles of all skeletons in the body coordinate system are required to be captured. Nevertheless, since the lower arm usually experiences a movement with fairly large range during the process of human motion, it is already representative to perform the activity sensing based on the angle profiles of the lower arm. For the angle profiles α, β and γ, according to the definition, they have the following relationship: (cos α) 2 + (cos β) 2 + (cos γ) 2 = 1. (5) Given any two values of the α, β and γ, the other one can be computed according to Eq.(5). However, it still has two candidate solutions according to the corresponding cosine value. Therefore, the arm-direction in the BCS can be uniquely determined via the three parameters hα, β, γi. For any specified meta-activity, while it is being performed, the angle profiles hα, β, γi are continuously changing. The meta-activity should have the following properties for any of the parameters hα, β, γi: 1) The variation range of any angle profile should be less than a threshold δ, e.g., 30◦ . 2) The variation trend of any angle profile should be monotonic, e.g., monotonically increasing or decreasing. 3) The time duration of the meta-activity should be less than a threshold t, e.g., 500ms. The first property implies that the moving range of the meta-activity should be small enough, the second property implies that the moving direction of the meta-activity should be monotonic, the third property implies that the time duration of the meta-activity should be limited, even if the moving range is still small enough. Therefore, for each dimension of the angle profiles hα, β, γi, we can uniformly divide the rotation range [0◦ , 360◦ ] into multiple sectors, while the angle of each sector is no greater than the threshold δ. In this way, we can use the specified
配 20 00 120 100 12 100 Dumbbell Curl Sit-Uo (a)Accleration measurements of Dumbbell Curl (b)Angle profiles of Dumbbell Curl (c)DTW distance in different complex activities Fig.4.The acceleration measurements and angle profiles in X,Y and Z-axes sector to depict the corresponding meta-activity in the spec- Data Acquisition and Preprocessing ified dimension.Moreover,considering the arm rotation can Raw Coordinate ngle Profile be anti-clock-wise or clock-wise,it is essential to further label Extraction Data each sector according to the rotation trend.Therefore,suppose Meta-Activity Segmentation and Classification the number of sectors is m,we can label each sector with a different ID from 0 to m-1 in an anti-clock-wise approach: Meta-Activity Meta-Activity for the ith sector,if the rotation direction is anti-clock-wise, then we label it with s;,otherwise,we label it with S,.Fig Complex Activity Recognition 5 shows an example of these meta-activity sectors,where Least Edit Distance-based Matching each sector has an angle of 30.These sectors are labeled from so to s11 if the rotation direction is anti-clock-wise, Fig.6.The system framework and they are labeled from So to S11 otherwise.In this way, BCS,by using the Direction Cosine method.To figure out we can use these discrete states rather than the continuous the orientation difference.i.e..the rotation matrix C between waveforms to represent the meta-activities.In comparison to BCS and GCS.before the human subject performs the complex the continuous waveform-based representation in the raw data activities,he/she is required to perform the following two level,this discrete state-based representation is based on the signal gestures in advance:1)Extend the arm to the front:let logic cognition of the human motion,which is more scalable the human subject extend his/her arm to the front of the body, to the inherent variances caused by user specific characters. the arm direction is consistent with the Y axis in the BCS:2) Drop the arm downward:let the human subject drop the arm s11 downward along his/her legs,the arm direction is opposite 81 10 s0:0°-30° to the Zo axis in the BCS.Fig.7(a)and Fig.7(b)shows an s1:30°-60 example of the two signal gestures,respectively.In this way, S2 S9s11:330°-360° by computing the corresponding vector of the arm direction S0:30°-0° in the GCS,we are able to figure out the rotation matrix C S1:60°-30° whatever the human subject is standing or lying on the floor. S11:360°-330° Glob Fig.5.The sectors to depict meta-activity in each dimension of angle profiles IV.SYSTEM DESIGN The overall system is composed of three major modules, as shown in Fig.6:Data Acquisition and Preprocessing takes the raw inertial measurements as input.It first performs the (a)Signal gesture 1:extend the arm (b)Signal gesture 2:drop the arm to the front downward coordinate transformation to transform the measurement from Fig.7.The signal gestures WCS to BCS.Then,it extracts the angle profiles and further 2)Angle Profiles Extraction:As aforementioned in Section split the series into separate complex activities.Meta-Activity III,take the smart watch as an example,we use the vector x, Segmentation and Classification segments a single complex i.e.,the direction of X axis in the WCS,to depict the arm activity into a series of meta-activities,and classifies the seg- direction in the BCS.Then,according to the arm vector x(t) mented meta-activities into corresponding categories.Complex at time t,we can extract the angle profiles (a(t),B(t),(t)) Activity Recognition performs activity recognition based on the over time in the BCS according to Eg.(4)-(7). sequences of meta-activities from the test complex activity,by 3)Segmentation:In practice,the human subject may con- leveraging the least edit distance-based matching scheme. tinuously perform a series of complex activities.Therefore,the recognition system should first split these series of complex A.Data Acquisition and Preprocessing activities into separate activities,then we can further identify 1)Coordinate Transformation:As mentioned in Section which activity pattern the current movement belongs to.As III,we can transform the measurement from the WCS to the human subject usually takes a short pause between two
0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d (a) Accleration measurements of Dumbbell Curl 0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d 0 20 40 60 80 100 120 140 0 0.5 1 a b c d (b) Angle profiles of Dumbbell Curl Dumbbell Curl Sit-Up 0 1 2 3 4 Acceleration Angle Profiles (c) DTW distance in different complex activities Fig. 4. The acceleration measurements and angle profiles in X, Y and Z-axes sector to depict the corresponding meta-activity in the specified dimension. Moreover, considering the arm rotation can be anti-clock-wise or clock-wise, it is essential to further label each sector according to the rotation trend. Therefore, suppose the number of sectors is m, we can label each sector with a different ID from 0 to m − 1 in an anti-clock-wise approach: for the ith sector, if the rotation direction is anti-clock-wise, then we label it with sj , otherwise, we label it with Sj . Fig. 5 shows an example of these meta-activity sectors, where each sector has an angle of 30◦ . These sectors are labeled from s0 to s11 if the rotation direction is anti-clock-wise, and they are labeled from S0 to S11 otherwise. In this way, we can use these discrete states rather than the continuous waveforms to represent the meta-activities. In comparison to the continuous waveform-based representation in the raw data level, this discrete state-based representation is based on the logic cognition of the human motion, which is more scalable to the inherent variances caused by user specific characters. 0 1 2 3 4 5 6 7 8 9 10 11 s0: 0º-30º s1: 30º-60º … s11: 330º-360º S0: 30º-0º S1: 60º-30º … S11: 360º-330º S0 S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 s s s s s s s s s s s s Fig. 5. The sectors to depict meta-activity in each dimension of angle profiles IV. SYSTEM DESIGN The overall system is composed of three major modules, as shown in Fig.6: Data Acquisition and Preprocessing takes the raw inertial measurements as input. It first performs the coordinate transformation to transform the measurement from WCS to BCS. Then, it extracts the angle profiles and further split the series into separate complex activities. Meta-Activity Segmentation and Classification segments a single complex activity into a series of meta-activities, and classifies the segmented meta-activities into corresponding categories. Complex Activity Recognition performs activity recognition based on the sequences of meta-activities from the test complex activity, by leveraging the least edit distance-based matching scheme. A. Data Acquisition and Preprocessing 1) Coordinate Transformation: As mentioned in Section III, we can transform the measurement from the WCS to Data Acquisition and Preprocessing Coordinate Transformation Angle Profile Extraction Segmentation Meta-Activity Segmentation and Classification Meta-Activity Segmentation Meta-Activity Classification Complex Activity Recognition Least Edit Distance-based Matching Raw Sensor Data Recognition Result Fig. 6. The system framework BCS, by using the Direction Cosine method. To figure out the orientation difference, i.e., the rotation matrix C 0 between BCS and GCS, before the human subject performs the complex activities, he/she is required to perform the following two signal gestures in advance: 1) Extend the arm to the front: let the human subject extend his/her arm to the front of the body, the arm direction is consistent with the Yb axis in the BCS; 2) Drop the arm downward: let the human subject drop the arm downward along his/her legs, the arm direction is opposite to the Zg axis in the BCS. Fig.7(a) and Fig.7(b) shows an example of the two signal gestures, respectively. In this way, by computing the corresponding vector of the arm direction in the GCS, we are able to figure out the rotation matrix C 0 , whatever the human subject is standing or lying on the floor. ! Global Coordinate System Arm Direction (a) Signal gesture 1: extend the arm to the front Global Coordinate System Arm Direction (b) Signal gesture 2: drop the arm downward Fig. 7. The signal gestures 2) Angle Profiles Extraction: As aforementioned in Section III, take the smart watch as an example, we use the vector xw, i.e., the direction of Xw axis in the WCS, to depict the arm direction in the BCS. Then, according to the arm vector xw(t) at time t, we can extract the angle profiles hα(t), β(t), γ(t)i over time in the BCS according to Eq. (4)-(7). 3) Segmentation: In practice, the human subject may continuously perform a series of complex activities. Therefore, the recognition system should first split these series of complex activities into separate activities, then we can further identify which activity pattern the current movement belongs to. As the human subject usually takes a short pause between two
adjacent complex activities.we use angle changes to detect the 2)Meta-Activity Classification:After the meta-activity seg- start and end of a complex activity.Specifically,we leverage a mentation.according to each dimension of the angle profiles. sliding window (which is set to 3s in our implementation)to the meta-activity should be further classified into one specific continuously store the recent angle profiles (a(t),B(t),(t)). sector based on its moving range and rotation direction.As For each angle profile,the system computes a derivative of the meta-activity depicts a process of movement ranging from these values,i.e.,the difference between the current and the a start angle 0 to an end angle 0e,it usually has some previous sample.When the difference from one or more angle fluctuations in the waveforms of the angle profiles.Hence, profiles exceeds a certain threshold (which is set to 10 in when performing meta-activity classification,it is essential to our implementation).the system detects the beginning of a consider the variation trend of angle profiles,instead of only complex activity;similarly,when the difference from almost the start and end angles.For example,a test meta-activity may all angle profiles falls bellow the same threshold,the system increase the angle profile slowly in part of the sector s5,and detects the end of a complex activity then increase the angle profile rapidly in part of the sector s6, B.Meta-Activity Segmentation and Classification so it should be classified to the sector s5 since its major angle profile varies in this area.Therefore,we leverage the method 1)Meta-Activity Segmentation:After preprocessing,we are of Dynamic Time Warping(DTW)to match a test meta-activity able to obtain the series of angle profiles ((t),B(t),(t))(tE to a corresponding sector by referring to the variation trend of [ts,tel)for a single complex activity,where ts and te are angle profiles.The procedure is as follows:In regard to any respectively the start time and the end time of the complex specific sector,considering the rotation range and direction,its activity.Then,it is essential to further segment the complex start angle profile is ns and its end angle profile is ne.Then, activity into a series of meta-activities,according to the series suppose the corresponding meta-activity is performed with a of angle profiles.Based on the properties of the meta-activity uniform speed in the time interval of At (At=500ms in our mentioned in Section III.B,we can derive the segmentation implementation),we can use a linear function f(t)to depict conditions for any of the angle profiles (i.e.,o,B or ) the template of this meta-activity,i.e.,f(t)=n+n.t. A straight-forward solution for meta-activity segmentation Then,given a test meta-activity with the angle profile n(t),and is to segment the series of angle profiles by checking all the the templates of meta-activities corresponding to the sectors, three dimensions of the angle profiles simultaneously.If the we can use a vector P with length I and a vector R with segmentation condition is satisfied in any of the dimensions, length l'to denote the test meta-activity's angle profile and the the entire series of angle profiles are segmented as a single specified template meta-activity's angle profile,respectively. meta-activity.However,this solution causes the series of For each pair of test meta-activity P and the template meta- angle profiles for a single meta-activity to be too fragmented activity Ry,we construct a distance matrix Dix as an input to after segmentation,since the series of angle profiles in the the DTW algorithm,where each element Di.j is defined as the other dimensions are not yet satisfied for the segmentation Euclidean distance between each pair of angle profiles Vp.i and condition to be a complete meta-activity,which is not suitable VR.j.The output of DTW is a warping path={,.,} for the following meta-activity classification.Therefore,we such that the distance d between the sequences is minimized: propose to perform the meta-activity segmentation over each argmindD().v().Hence.given the test meta- dimension of the angle profiles in a separate approach.For activity,we can enumerate all template meta-activities and each dimension,we leverage a sliding window (which is set leverage DTW to compute the corresponding distance.We then to 500ms in our implementation)to continuously store the select the template meta-activity with the smallest distance as recent angle profiles.Our solution scans the series of angle the classification result. profiles to verify if any of the segmentation condition is satisfied,we then segment the angle profile series as a single C.Complex Activity Recognition meta-activity in the corresponding dimension.After the meta- After the meta-activity classification,each complex activity activity segmentation,we can obtain a separate segmentation can be decomposed into a sequence of meta-activities,respec- for each dimension of the angle profiles.Fig.8 shows an tively,in regard to the three dimensions of angle profiles. example of the meta-activity segmentation for the activity We call them meta-activity profiles of the complex activity Dumbbell Curl in the three dimensions of the angle profiles. In Table I,we show four example meta-activity profiles for 400 the specified complex activities. 820 In this paper,we leverage the Least Edit Distance-based 00 120 140 matching scheme (LED)to perform the complex activity 400 82 recognition.LED compares the test complex activity against the template complex activities in regard to the corresponding 00 120 "40 00 meta-activity profiles.It computes the least edit distance be- tween each pair of test complex activity and template complex 120 140 activity,and select the template complex activity with the The index of samplings(50 samples/s) smallest distance as the matching result.By referring to the Fig.8.An example of the meta-activity segmentation edit distance [7]for measuring the difference between two
adjacent complex activities, we use angle changes to detect the start and end of a complex activity. Specifically, we leverage a sliding window (which is set to 3s in our implementation) to continuously store the recent angle profiles hα(t), β(t), γ(t)i. For each angle profile, the system computes a derivative of these values, i.e., the difference between the current and the previous sample. When the difference from one or more angle profiles exceeds a certain threshold (which is set to 10◦ in our implementation), the system detects the beginning of a complex activity; similarly, when the difference from almost all angle profiles falls bellow the same threshold, the system detects the end of a complex activity. B. Meta-Activity Segmentation and Classification 1) Meta-Activity Segmentation: After preprocessing, we are able to obtain the series of angle profiles hα(t), β(t), γ(t)i(t ∈ [ts, te]) for a single complex activity, where ts and te are respectively the start time and the end time of the complex activity. Then, it is essential to further segment the complex activity into a series of meta-activities, according to the series of angle profiles. Based on the properties of the meta-activity mentioned in Section III.B, we can derive the segmentation conditions for any of the angle profiles (i.e., α, β or γ). A straight-forward solution for meta-activity segmentation is to segment the series of angle profiles by checking all the three dimensions of the angle profiles simultaneously. If the segmentation condition is satisfied in any of the dimensions, the entire series of angle profiles are segmented as a single meta-activity. However, this solution causes the series of angle profiles for a single meta-activity to be too fragmented after segmentation, since the series of angle profiles in the other dimensions are not yet satisfied for the segmentation condition to be a complete meta-activity, which is not suitable for the following meta-activity classification. Therefore, we propose to perform the meta-activity segmentation over each dimension of the angle profiles in a separate approach. For each dimension, we leverage a sliding window (which is set to 500ms in our implementation) to continuously store the recent angle profiles. Our solution scans the series of angle profiles to verify if any of the segmentation condition is satisfied, we then segment the angle profile series as a single meta-activity in the corresponding dimension. After the metaactivity segmentation, we can obtain a separate segmentation for each dimension of the angle profiles. Fig. 8 shows an example of the meta-activity segmentation for the activity Dumbbell Curl in the three dimensions of the angle profiles. s1 s2 s3 s4 s5 S6 S4 s8 S2 S1 S1 S0 s0 s1 s2 S8 S2 s2 s2 s9 s10 s11 s0 s1 Fig. 8. An example of the meta-activity segmentation 2) Meta-Activity Classification: After the meta-activity segmentation, according to each dimension of the angle profiles, the meta-activity should be further classified into one specific sector based on its moving range and rotation direction. As the meta-activity depicts a process of movement ranging from a start angle θs to an end angle θe, it usually has some fluctuations in the waveforms of the angle profiles. Hence, when performing meta-activity classification, it is essential to consider the variation trend of angle profiles, instead of only the start and end angles. For example, a test meta-activity may increase the angle profile slowly in part of the sector s5, and then increase the angle profile rapidly in part of the sector s6, so it should be classified to the sector s5 since its major angle profile varies in this area. Therefore, we leverage the method of Dynamic Time Warping (DTW) to match a test meta-activity to a corresponding sector by referring to the variation trend of angle profiles. The procedure is as follows: In regard to any specific sector, considering the rotation range and direction, its start angle profile is ηs and its end angle profile is ηe. Then, suppose the corresponding meta-activity is performed with a uniform speed in the time interval of ∆t (∆t=500ms in our implementation), we can use a linear function f(t) to depict the template of this meta-activity, i.e., f(t) = ηs + ηe−ηs ∆t · t. Then, given a test meta-activity with the angle profile η(t), and the templates of meta-activities corresponding to the sectors, we can use a vector Pl with length l and a vector Rl 0 with length l 0 to denote the test meta-activity’s angle profile and the specified template meta-activity’s angle profile, respectively. For each pair of test meta-activity Pl and the template metaactivity Rl 0 , we construct a distance matrix Dl×l 0 as an input to the DTW algorithm, where each element Di,j is defined as the Euclidean distance between each pair of angle profiles VP,i and VR,j . The output of DTW is a warping path π = {π1, ..., πk} such that the distance d between the sequences is minimized: argminπ dπ = Pk i=1 Dx(π),y(π) . Hence, given the test metaactivity, we can enumerate all template meta-activities and leverage DTW to compute the corresponding distance. We then select the template meta-activity with the smallest distance as the classification result. C. Complex Activity Recognition After the meta-activity classification, each complex activity can be decomposed into a sequence of meta-activities, respectively, in regard to the three dimensions of angle profiles. We call them meta-activity profiles of the complex activity. In Table I, we show four example meta-activity profiles for the specified complex activities. In this paper, we leverage the Least Edit Distance-based matching scheme (LED) to perform the complex activity recognition. LED compares the test complex activity against the template complex activities in regard to the corresponding meta-activity profiles. It computes the least edit distance between each pair of test complex activity and template complex activity, and select the template complex activity with the smallest distance as the matching result. By referring to the edit distance [7] for measuring the difference between two
Complex Activity Meta-Activity Profiles Dumbbell a:(Sg,S8,S7,S5,s7,s8） After that,for any two complex activities ci and cj.we add Triceps B:(s8,S8》 the distances from all three dimensions together,and obtain the Extension Y:(s5,s6,S7) overall distance between the two complex activities as follows: Upright a:(s0,s2,s3,53,S2,S1】 Barbell B:(s10,S10,S11,S10） Row Y:(s3,s8,S8) Lee=VLe(a)()+I()(+)) (8) Dumbbell a:(s0,82,s3,sg,810 Lateral 3:(s2》 Therefore,given a test complex activity c.we enumerate all Raise Y:(S2,S1,S0,s11,s0,s1》 template meta-activity profiles of all complex activities ciEC Butter a:(82》 Fly B:(S2,S1,S0,S11,s1,s2)》 and compute their distance Le then we select the category y:(s0,s2,S2,S1,S0】 of the template complex activity with the least distance as the TABLE I recognition result. EXAMPLE META-ACTIVITY PROFILES FOR THE COMPLEX ACTIVITIES V.PERFORMANCE EVALUATION sequences of strings,we leverage this term to denote the A.Experimental Setup difference between two sequences of meta-activity profiles. We have implemented a prototype system using the android In order to compute the edit distance between two pairs of phone(SAMSUNG Galaxy S5)2,which is attached to the wrist complex activities,it is essential to first consider the distance of the human subject,as shown in Fig.9.The android phone between two meta-activities.As aforementioned in Section III, is embedded with inertial sensors including accelerometers for each dimension of the angle profiles,the meta-activity and magnetometers.The lower-arm direction is consistent with is a process which is performed in a specified sector with the Y-axis of the smart phone's local coordinate system.In a specified rotation direction.When considering the distance the experiment,we let 10 volunteers perform 10 categories between two meta-activities,we should take these two issues of complex activities,they have different heights,genders, into consideration.i.e..the distance between sectors and the and ages.For each category of complex activity,20 sam- distance between rotation directions.Considering the distance ples of inertial measurements are collected for each subject. between sectors,assume the number of sectors is m,for any In order to evaluate the performance for user-independent two meta-activities mi and mj,suppose their corresponding activity sensing,we leverage the n-fold cross-validation as sector numbers are respectively si and sj(0<si<sj<m). follows:for each round of evaluation,we select one human the distance between them is defined as follows: subject as the test case,and obtain the template profiles from d.(mi;mj)=min{(sj-si)modm,(si-sj+m)modm}. n-1 of the remaining human subjects.We then evaluate the recognition accuracy and time delay for the three solutions: The distance is the minimum distance between the two sectors 1)Acceleration-based Matching (AM):It uses the DTW to si and s;either clockwise or anti-clockwise.Considering the perform waveform-based matching in terms of the acceleration distance between rotation directions(clock-wise or anti-clock- measurements.2)Angle Profiles-based Matching (APM):It wise),for any two meta-activities mi and mj,if they have the uses the DTW to perform waveform-based matching in terms same rotation direction,then we set the distance dr(mi,mj) of the angle profiles.3)Meta-Activity Recognition(MAR):It to 0.Otherwise,we set the distance dr(mi,mj)to (n= uses the least edit distance-based matching in terms of the m/4 in our implementation).Hence,considering the above meta-activity profiles. two issues,the distance between two meta-activities m;and mj is as follows: d(m,m）=d,(m,mj）+d,(m,m) (6) Therefore,for each dimension of the angles profiles,a complex activity ciC can be depicted as a sequence of the meta-activities,i.e,c=(mi,…,m）,where mj∈M. Then,for a specified dimension,considering any two com- Fig.9.Experimental Setup plex activities,e.g.,a and b,we can compute their distance B.Parameter Selection Lab(lal,b)by referring to the Levenshtein distance [7]: For the meta-activity recognition,the angle of a meta- max(i,j)×4 if min(i,j)=0, activity sector,i.e.,6,is very crucial to the performance in La.b(i,j)= La.b(i-1,j)+4 (7) terms of recognition accuracy and time efficiency.It directly min La.6(i,j-1)+u otherwise. determines the number of meta-activities within a specified La.b(i-1,j-1)+d(ai;bj) complex activity.Therefore,we conduct experiments to eval- where La.(i,j)is the distance between the first i meta- uate the performance with different values of 6.We set the activities of a and the first j meta-activities of b.u is the av- number of human subjects in template construction to 5. erage distance between any two meta-activities (u=0.75 x m in our implementation),and d(ai,bj)is the distance between 2As COTS smart watches are still not embedded with magnetometers to help build the body coordinate system,so in this paper we choose to use the the ith meta-activity of a and the jth meta-activity of b. android phone as the testing wearable devices
Complex Activity Meta-Activity Profiles Dumbbell α : hS9, S8, S7, S5, s7, s8i Triceps β : hs8, S8i Extension γ : hs5, s6, S7i Upright α : hs0, s2, s3, S3, S2, S1i Barbell β : hs10, S10, S11, S10i Row γ : hs3, s8, S8i Dumbbell α : hs0, s2, s3, s9, s10i Lateral β : hs2i Raise γ : hS2, S1, S0, s11, s0, s1i Butter α : hs2i Fly β : hS2, S1, S0, S11, s1, s2i γ : hs0, s2, S2, S1, S0i TABLE I EXAMPLE META-ACTIVITY PROFILES FOR THE COMPLEX ACTIVITIES sequences of strings, we leverage this term to denote the difference between two sequences of meta-activity profiles. In order to compute the edit distance between two pairs of complex activities, it is essential to first consider the distance between two meta-activities. As aforementioned in Section III, for each dimension of the angle profiles, the meta-activity is a process which is performed in a specified sector with a specified rotation direction. When considering the distance between two meta-activities, we should take these two issues into consideration, i.e., the distance between sectors and the distance between rotation directions. Considering the distance between sectors, assume the number of sectors is m, for any two meta-activities mi and mj , suppose their corresponding sector numbers are respectively si and sj (0 ≤ si ≤ sj < m), the distance between them is defined as follows: ds(mi , mj ) = min{(sj − si)modm,(si − sj + m)modm}. The distance is the minimum distance between the two sectors si and sj either clockwise or anti-clockwise. Considering the distance between rotation directions (clock-wise or anti-clockwise), for any two meta-activities mi and mj , if they have the same rotation direction, then we set the distance dr(mi , mj ) to 0. Otherwise, we set the distance dr(mi , mj ) to Ω (Ω = m/4 in our implementation). Hence, considering the above two issues, the distance between two meta-activities mi and mj is as follows: d(mi , mj ) = ds(mi , mj ) + dr(mi , mj ). (6) Therefore, for each dimension of the angles profiles, a complex activity ci ∈ C can be depicted as a sequence of the meta-activities, i.e., ci = hmj1 , · · · , mjk i, where mj ∈ M. Then, for a specified dimension, considering any two complex activities, e.g., a and b, we can compute their distance La,b(|a|, |b|) by referring to the Levenshtein distance [7]: La,b(i, j) = max(i, j) × µ if min(i, j) = 0, min La,b(i − 1, j) + µ La,b(i, j − 1) + µ otherwise. La,b(i − 1, j − 1) + d(ai , bj ) (7) where La,b(i, j) is the distance between the first i metaactivities of a and the first j meta-activities of b, µ is the average distance between any two meta-activities (µ = 0.75×m in our implementation), and d(ai , bj ) is the distance between the ith meta-activity of a and the jth meta-activity of b. After that, for any two complex activities ci and cj , we add the distances from all three dimensions together, and obtain the overall distance between the two complex activities as follows: Lci,cj = q L 2 ci(α),cj (α) + L 2 ci(β),cj (β) + L 2 ci(γ),cj (γ) . (8) Therefore, given a test complex activity ci , we enumerate all template meta-activity profiles of all complex activities cj ∈ C and compute their distance Lci,cj , then we select the category of the template complex activity with the least distance as the recognition result. V. PERFORMANCE EVALUATION A. Experimental Setup We have implemented a prototype system using the android phone (SAMSUNG Galaxy S5)2 , which is attached to the wrist of the human subject, as shown in Fig.9. The android phone is embedded with inertial sensors including accelerometers and magnetometers. The lower-arm direction is consistent with the Y -axis of the smart phone’s local coordinate system. In the experiment, we let 10 volunteers perform 10 categories of complex activities, they have different heights, genders, and ages. For each category of complex activity, 20 samples of inertial measurements are collected for each subject. In order to evaluate the performance for user-independent activity sensing, we leverage the n-fold cross-validation as follows: for each round of evaluation, we select one human subject as the test case, and obtain the template profiles from n − 1 of the remaining human subjects. We then evaluate the recognition accuracy and time delay for the three solutions: 1) Acceleration-based Matching (AM): It uses the DTW to perform waveform-based matching in terms of the acceleration measurements. 2) Angle Profiles-based Matching (APM): It uses the DTW to perform waveform-based matching in terms of the angle profiles. 3) Meta-Activity Recognition (MAR): It uses the least edit distance-based matching in terms of the meta-activity profiles. Z-axis Y-axis X-axis Fig. 9. Experimental Setup B. Parameter Selection For the meta-activity recognition, the angle of a metaactivity sector, i.e., δ, is very crucial to the performance in terms of recognition accuracy and time efficiency. It directly determines the number of meta-activities within a specified complex activity. Therefore, we conduct experiments to evaluate the performance with different values of δ. We set the number of human subjects in template construction to 5. 2As COTS smart watches are still not embedded with magnetometers to help build the body coordinate system, so in this paper we choose to use the android phone as the testing wearable devices
150 0 50 20 5° 10° 20° 30° 15 The number of human subjects in template construction (c)Accuracy for different number of training samples A1 总AAMA5帖A7随A1O AI2EAA5AGA78超A1O A1点9 A3 AA A5 AS A7A A A10 A100000051000000a00.00.00.0 A1U0000.i000a0a000a00000 At1.创Q0a0a0a0a0ai0ai0a000网 3000 200的00000.110000000.00.00.00.0 0.0100.00a80a00000m 30.0的0.0时0s000020000.030.00.B00.0 A3-0.600.601600.0Q600.00.00Q000000 A3a.00Q01.00.00000a00000a.000.00 A40050.010.000940000000.600.600.00.0 A4-0.00.600.01.60a.60a.600.00000.000.00- Ma.00Q.000001.0a.0Q0a00a0的a.000.00 2000 A50.0的0.0的00105098a000.00.00.00.0 A50.000.00010000的0060Q.0600000 A54.00意.000000.0100000000的0000.00 50.200500的0能00006的a00.00.50.07 A80.30.00.0a1208053aa0019012, 8.01000050500的a0的ai4024 A7040000000000001700.230.20.0 A70.00.00.300008002010a0mQ000m A7.00 .00.00.00 0.00 0.010.00 0.03 0.00 91000 80.000.000.000010000000.0的0.30.130.0 A80.00.00.00.a00.800.00001.m00000 h8a.00Q.0000000a0a0的1D的a.00000 A9a.1100的000ag0000i00.140.00.470.1 A9-0.0.0.200.200.的0.6005401Q0的0g0.0- 9a.5a.00.000.t0016112001a650.01- A10也140500的0a000236000.0010.42 1000014090990509900m0m29 The number of human subjects in templates (d)Confusion matrix for AM (e)Confusion matrix for APM (f)Confusion matrix for MAR (g)Time delay for different solutions Fig.10.The experiment results We first evaluate the performance in terms of recognition accuracy,as shown in Fig.10(c).It is found that,in almost accuracy when the angle6 is varied from5°to45°，as shown all situations,MAR achieves the best performance whereas AM in Fig.10(a).It is found that all the recognition accuracies are achieves the worst performance.Specifically,when the number greater than 83%when the angle is varied from 5 to 45. of human subjects in template construction is 1,the number The highest accuracy is achieved when the angle is set to 10, of training samples is small.AM achieves poor accuracy of whereas the lowest accuracy is achieved when the angle is 61%as it lacks enough templates for accurate matching. set to 45.The reason is that,when the angle is fairly large,APM leverages the angle profiles to mitigate the impact of e.g.,45,the granularity of the meta-activity is too coarse to variances in the raw inertial measurement,thus it enhances depict the movement of human subjects,thus it leads to many the accuracy from 61%to 81%.Due to the character of mismatches in activity recognition.However,when the angle logical cognition,MAR further improves the accuracy to 87% is fairly small,e.g.,5,the granularity of the meta-activity even if the training samples are so limited.This implies that is too fine to tolerate the detailed deviations due to human- our meta-activity recognition achieve rather good performance specific characters,thus the performance is also degraded in for user-independent recognition while requiring lightweight- comparison to the optimum case.Therefore,the parameter of training.As the number of templates increases,the accuracies the sector angle should be carefully selected for improving the of the three solutions are all increasing to a close value of performance in recognition accuracy. 92%.Nevertheless.MAR always achieves the least variances in We then evaluate the performance in terms of time efficiency recognition accuracy since it leads to very stable performance. when the angle6 is varied from5°to45°，as shown in Fig. 2)The matching ratios among multiple activities:We fur- 10(b).It is found that,as the angle 6 increases from 5 to ther investigate the confusion matrices for the three solutions, 30,the average time delay rapidly decreases from 145ms to as shown from Fig.10(d)to Fig.10(f).We set the number 21ms.The reduced time delay is caused by the increasing of human subjects in template construction to 2,and the granularity of the meta-activity,which reduces the processing activities are listed from A1 to A10 according to the order in time cost.However,when the angle o further increases to 45 Fig.1.According to the matching results in the three confusion the average time delay slightly increases to 49ms,as the time matrices,it is found that APM is able to reduce most of the delay for the meta-activity classification increases due to the mismatches caused by AM.so APM achieves the recognition increased size of input to the DTW algorithm.Therefore,to ratio of 100%for most activities.However,APM still fails to achieve an appropriate trade off between the accuracy and time accurately recognize some activities such as A6,Ag and A10. efficiency,in the following experiment,we set the angle o to since these activities usually have larger movement ranges 10 in MAR to achieve the optimized performance. and more movement variations in details.Fortunately,MAR is able to further reduce these mismatches and improve the C.Evaluate the Recognition Accuracy recognition accuracy to a fairly high level.Moreover,MAR 1)Sensitivity to the number of training samples:Since achieves the least variances in recognizing multiple activities we aim to achieve the lightweight-training recognition,we in comparison to the other two solutions. require the number of training samples to be sa small as possible.Therefore,as we collect 20 samples from each human D.Evaluate the Time Efficiency subject to build the templates for each complex activity,we We further evaluate the time delay of processing the activity vary the number of human subjects involved in the template sensing,respectively,for AM,APM and MAR.We vary the construction from I to 8,and evaluate the average recognition number of human subjects in template construction from I to
The angle of the meta-activity sector 5 ° 10 ° 20 ° 30 ° 45 ° Recognition accuracy 0.5 0.6 0.7 0.8 0.9 1 (a) Accuracy for different sector angles The angle of the meta-activity sector 5° 10° 20° 30° 45° Time delay(ms) 0 50 100 150 (b) Time delay for different sector angles The number of human subjects in template construction 1 2 4 8 Recognition Accuracy 0 0.2 0.4 0.6 0.8 1 AM APM MAR (c) Accuracy for different number of training samples 0.49 0.00 0.00 0.05 0.00 0.12 0.04 0.00 0.11 0.14 0.00 0.89 0.01 0.01 0.00 0.05 0.00 0.00 0.00 0.05 0.00 0.00 0.95 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.51 0.11 0.00 0.94 0.00 0.02 0.00 0.01 0.18 0.02 0.00 0.00 0.02 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.68 0.01 0.00 0.10 0.36 0.00 0.00 0.03 0.00 0.00 0.00 0.70 0.03 0.14 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.23 0.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.02 0.13 0.47 0.01 0.00 0.00 0.00 0.00 0.00 0.07 0.00 0.00 0.01 0.42 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 (d) Confusion matrix for AM 1.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.14 0.00 0.00 1.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.12 0.00 0.00 0.03 0.08 0.00 0.00 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.25 0.00 0.00 0.00 0.00 0.00 0.53 0.00 0.00 0.54 0.03 0.00 0.00 0.00 0.00 0.00 0.01 1.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.19 0.00 0.00 0.39 0.00 0.00 0.00 0.00 0.00 0.00 0.12 0.00 0.00 0.00 0.49 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 (e) Confusion matrix for APM 1.00 0.01 0.00 0.00 0.00 0.01 0.00 0.00 0.05 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.05 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.05 0.00 0.00 0.01 0.02 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.57 0.01 0.00 0.16 0.03 0.00 0.00 0.00 0.00 0.00 0.00 0.96 0.00 0.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.14 0.03 0.00 0.65 0.00 0.00 0.00 0.00 0.00 0.00 0.24 0.00 0.00 0.01 0.90 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 (f) Confusion matrix for MAR The number of human subjects in templates 1 2 4 8 Time delay(ms) 0 1000 2000 3000 AM APM MAR (g) Time delay for different solutions Fig. 10. The experiment results We first evaluate the performance in terms of recognition accuracy when the angle δ is varied from 5 ◦ to 45◦ , as shown in Fig. 10(a). It is found that all the recognition accuracies are greater than 83% when the angle is varied from 5 ◦ to 45◦ . The highest accuracy is achieved when the angle is set to 10◦ , whereas the lowest accuracy is achieved when the angle is set to 45◦ . The reason is that, when the angle is fairly large, e.g., 45◦ , the granularity of the meta-activity is too coarse to depict the movement of human subjects, thus it leads to many mismatches in activity recognition. However, when the angle is fairly small, e.g., 5 ◦ , the granularity of the meta-activity is too fine to tolerate the detailed deviations due to humanspecific characters, thus the performance is also degraded in comparison to the optimum case. Therefore, the parameter of the sector angle should be carefully selected for improving the performance in recognition accuracy. We then evaluate the performance in terms of time efficiency when the angle δ is varied from 5 ◦ to 45◦ , as shown in Fig. 10(b). It is found that, as the angle δ increases from 5 ◦ to 30◦ , the average time delay rapidly decreases from 145ms to 21ms. The reduced time delay is caused by the increasing granularity of the meta-activity, which reduces the processing time cost. However, when the angle δ further increases to 45◦ , the average time delay slightly increases to 49ms, as the time delay for the meta-activity classification increases due to the increased size of input to the DTW algorithm. Therefore, to achieve an appropriate trade off between the accuracy and time efficiency, in the following experiment, we set the angle δ to 10◦ in MAR to achieve the optimized performance. C. Evaluate the Recognition Accuracy 1) Sensitivity to the number of training samples: Since we aim to achieve the lightweight-training recognition, we require the number of training samples to be sa small as possible. Therefore, as we collect 20 samples from each human subject to build the templates for each complex activity, we vary the number of human subjects involved in the template construction from 1 to 8, and evaluate the average recognition accuracy, as shown in Fig. 10(c). It is found that, in almost all situations, MAR achieves the best performance whereas AM achieves the worst performance. Specifically, when the number of human subjects in template construction is 1, the number of training samples is small. AM achieves poor accuracy of 61% as it lacks enough templates for accurate matching. APM leverages the angle profiles to mitigate the impact of variances in the raw inertial measurement, thus it enhances the accuracy from 61% to 81%. Due to the character of logical cognition, MAR further improves the accuracy to 87% even if the training samples are so limited. This implies that our meta-activity recognition achieve rather good performance for user-independent recognition while requiring lightweighttraining. As the number of templates increases, the accuracies of the three solutions are all increasing to a close value of 92%. Nevertheless, MAR always achieves the least variances in recognition accuracy since it leads to very stable performance. 2) The matching ratios among multiple activities: We further investigate the confusion matrices for the three solutions, as shown from Fig.10(d) to Fig.10(f). We set the number of human subjects in template construction to 2, and the activities are listed from A1 to A10 according to the order in Fig.1. According to the matching results in the three confusion matrices, it is found that APM is able to reduce most of the mismatches caused by AM, so APM achieves the recognition ratio of 100% for most activities. However, APM still fails to accurately recognize some activities such as A6, A9 and A10, since these activities usually have larger movement ranges and more movement variations in details. Fortunately, MAR is able to further reduce these mismatches and improve the recognition accuracy to a fairly high level. Moreover, MAR achieves the least variances in recognizing multiple activities in comparison to the other two solutions. D. Evaluate the Time Efficiency We further evaluate the time delay of processing the activity sensing, respectively, for AM, APM and MAR. We vary the number of human subjects in template construction from 1 to
8 and evaluate the corresponding time delay.It is found that,Our solution extracts the angle profiles to depict the angle in all situations,MAR achieves much smaller time delay (all variation of limb movement in the consistent body coordinate less than 45ms)than AM and APM.Moreover,as the number system.It further extracts the meta-activity profiles to depict of human subjects in template construction increases from 1 to the sequence of small range activities in the complex activity. 8,the time delay of AM and APM increases rapidly,whereas By leveraging the least edit distance-based matching scheme, the time delay of MAR keeps fairly stable.The reason is as the experiment results shows that our solution achieves an follows:As both AM and APM uses the DTW for matching,it average accuracy of 92%for user-independent activity sensing. requires a large amount of time to process the measurements ACKNOWLEDGMENTS with a small granularity in the raw data level,however,MAR This work is supported in part by National Natural Science processes the measurements with a much larger granularity in Foundation of China under Grant Nos.61472185,61373129, the meta-activity level.It takes most of the processing time 61321491,61502224;JiangSu Natural Science Foundation, on the meta-activity segmentation rather than matching.Thus, No.BK20151390.This work is partially supported by Col- MAR achieves the best time efficiency in tens of milliseconds. laborative Innovation Center of Novel Software Technology VI.RELATED WORK and Industrialization REFERENCES Wearable Device.Recent researches consider leveraging the inertial sensors embedded in wearable devices to detect [1]A.Parate.M.C.Chiu.C.Chadowitz.D.Ganesan.and E.Kalogerakis and monitor the user's activities [1],[8]-[13].Wrist mounted Risq:Recognizing smoking gestures with inertial sensors on a wristband In Proceedings of ACM MobiSys.2014. inertial sensors are widely used for arm-based activity sensing [2]C.Karatas,L.Liu,H.Li,J.Liu,Y.Wang,S.Tan,J.Yang,Y.Chen [1].[2].RistQ [1]leverages the accelerations from a wrist M.Gruteser,and R.Martin.Leveraging wearables for steering and driver tracking.In Proceedings of IEEE INFOCOM,2016. strap to detect and recognize smoking gestures.Karatas et al. [3]S.Jain,C.Borgiattino,Y.Ren,M.Gruteser,Y.Chen,and C.Fabiana [2]uses wrist mounted inertial sensors to track steering wheel Chiasserini.Lookup:Enabling pedestrian safety services via shoe usage and angle.Foot-mounted inertial sensors are leveraged sensing.In Proceedings of ACM MobiSys,2015. for indoor localization by sensing the patterns of footsteps [4]S.Butterworth.On the theory of filter amplifiers.Wireless Engineer, 7(6:536-541.1930. [12],[13].LookUp [3]uses shoe-mounted inertial sensors for [5]M.R.Spiegel,S.Lipschutz,and D.Spellman.Vector analysis.Schatmns location classification based on surface gradient profile and Outlines(2nd ed.).pages 15-25.2009. step patterns.Robertson et al.[12]proposes an approach for [6]K.Shoemake.Animating rotation with quaternion curves.Computer Graphics,19(3):245-254.1985. simultaneous mapping and localization for pedestrians based [7]G.Navarro.A guided tour to approximate string matching.ACM on odometry with foot mounted inertial sensors. Computing Surveys,33(11):31-88,2001. [8]S.Consolvo,D.W.McDonald,T.Toscos.M.Y.Chen,J.Froehlich. Wireless Signals.Another branch of activity recognition B.Harrison,P.Klasnja.A.LaMarca,L.LeGrand,and R.Libby.Activity solutions exploit the change of wireless signals (including sensing in the wild:A field trial of ubifit garden.In Proceedings of ACM WiFi signals,RF-signals,etc.)incurred by the human activities CHL.2008. [9]K.-H.Chang.M.Y.Chen,and J.Canny.Tracking free-weight exercises. [14]-[19].FEMO [14]provides a free-weight exercise moni- In Proceedings of ACM UbiComp,2007 toring scheme by attaching RFID tags on the dumbbells and [10]A.Khany,S.Mellory,E.BerlinN,R.Thompsony,R.McNaneyy, leveraging the Doppler shift profile of the reflected backscatter P.Oliviery,and T.Plotzy.Beyond activity recognition:Skill assessment from accelerometer data.In Proceedings of ACM UbiComp,2015. signals for activity recognition.Wang et al.[15]propose a [11]N.Roy,H.Wang,and R.R.Choudhury.I am a smartphone and I can CSI based human activity recognition and monitoring system, tell my user's walking direction.In Proceedings of MobiSys,2014. by quantitatively building the correlation between CSI value [12]P.Robertson.M.Angermann,and B.Krach.Simultaneous localization and mapping for pedestrians using only foot-mounted inertial sensors. dynamics and a specific human activity.E-eyes [16]presents In Proceedings of ACM UbiComp,2009 device-free location-oriented activity identification at home [13]O.Woodman and R.Harle.Pedestrian localization for indoor environ- through the use of fine-grained WiFi signatures.RF-IDraw ments.In Proceedings of ACM UbiComp.2008. [17]can infer a humans writing by tracking a passive RFID [14]H.Ding.L.Shangguan,Z.Yang.J.Han,Z.Zhou,P.Yang.W.Xi,and J.Zhao.Femo:A platform for free-weight exercise monitoring with tag attached to his/her fingers. rfids.In Proceedings of ACM SenSys,2015. However,most of the above activity recognition schemes [15]W.Wang.A.X.Liu,M.Shahzad,K.Ling,and S.Lu.Understanding and modeling of wifi signal based human activity recognition.In leverage the traditional waveform-based matching to process Proceedings of ACM MobiCom,2015. the inertial measurement/wireless signals in the raw data [16]Y.Wang.J.Liu.Y.Chen.M.Gruteser,J.Yang,and H.Liu.E-eyes: level.In this paper,we propose the meta-activity recogni- device-free location-oriented activity identification using fine-grained tion,which belongs to logic cognition-based activity sensing. wifi signatures.In Proceedings of ACM MOBICOM,2014. [17]J.Wang.D.Vasisht,and D.Katabi.RF-IDraw:Virtual touch screen in Our approach achieves lightweight-training recognition,which the air using RF signals.In Proc.of ACM SIGCOMM,2014. requires a small quantity of training samples to build the [18]X.Zheng,J.Wang.L.Shangguan.Z.Zhou,and Y.Liu.Smokey: Ubiquitous smoking detection with commercial wifi infrastructures.In templates,and user-independent recognition,which requires Proceedings of IEEE INFOCOM,2016. no training from the specific user. [19]Q.Pu,S.Gupta.S.Gollakota,and S.Patel.Whole-home gesture VII.CONCLUSION recognition using wireless signals.In Proceedings of ACM MOBICOM. 2013. In this paper,we propose a wearable approach for logic cognition-based activity sensing scheme in the logical repre- sentation level,by leveraging the meta-activity recognition
8 and evaluate the corresponding time delay. It is found that, in all situations, MAR achieves much smaller time delay (all less than 45ms) than AM and APM. Moreover, as the number of human subjects in template construction increases from 1 to 8, the time delay of AM and APM increases rapidly, whereas the time delay of MAR keeps fairly stable. The reason is as follows: As both AM and APM uses the DTW for matching, it requires a large amount of time to process the measurements with a small granularity in the raw data level, however, MAR processes the measurements with a much larger granularity in the meta-activity level. It takes most of the processing time on the meta-activity segmentation rather than matching. Thus, MAR achieves the best time efficiency in tens of milliseconds. VI. RELATED WORK Wearable Device. Recent researches consider leveraging the inertial sensors embedded in wearable devices to detect and monitor the user’s activities [1], [8]–[13]. Wrist mounted inertial sensors are widely used for arm-based activity sensing [1], [2]. RistQ [1] leverages the accelerations from a wrist strap to detect and recognize smoking gestures. Karatas et al. [2] uses wrist mounted inertial sensors to track steering wheel usage and angle. Foot-mounted inertial sensors are leveraged for indoor localization by sensing the patterns of footsteps [12], [13]. LookUp [3] uses shoe-mounted inertial sensors for location classification based on surface gradient profile and step patterns. Robertson et al. [12] proposes an approach for simultaneous mapping and localization for pedestrians based on odometry with foot mounted inertial sensors. Wireless Signals. Another branch of activity recognition solutions exploit the change of wireless signals (including WiFi signals, RF-signals, etc.) incurred by the human activities [14]–[19]. FEMO [14] provides a free-weight exercise monitoring scheme by attaching RFID tags on the dumbbells and leveraging the Doppler shift profile of the reflected backscatter signals for activity recognition. Wang et al. [15] propose a CSI based human activity recognition and monitoring system, by quantitatively building the correlation between CSI value dynamics and a specific human activity. E-eyes [16] presents device-free location-oriented activity identification at home through the use of fine-grained WiFi signatures. RF-IDraw [17] can infer a humans writing by tracking a passive RFID tag attached to his/her fingers. However, most of the above activity recognition schemes leverage the traditional waveform-based matching to process the inertial measurement/wireless signals in the raw data level. In this paper, we propose the meta-activity recognition, which belongs to logic cognition-based activity sensing. Our approach achieves lightweight-training recognition, which requires a small quantity of training samples to build the templates, and user-independent recognition, which requires no training from the specific user. VII. CONCLUSION In this paper, we propose a wearable approach for logic cognition-based activity sensing scheme in the logical representation level, by leveraging the meta-activity recognition. Our solution extracts the angle profiles to depict the angle variation of limb movement in the consistent body coordinate system. It further extracts the meta-activity profiles to depict the sequence of small range activities in the complex activity. By leveraging the least edit distance-based matching scheme, the experiment results shows that our solution achieves an average accuracy of 92% for user-independent activity sensing. ACKNOWLEDGMENTS This work is supported in part by National Natural Science Foundation of China under Grant Nos. 61472185, 61373129, 61321491, 61502224; JiangSu Natural Science Foundation, No. BK20151390. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. REFERENCES [1] A. Parate, M. C. Chiu, C. Chadowitz, D. Ganesan, and E. Kalogerakis. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In Proceedings of ACM MobiSys, 2014. [2] C. Karatas, L. Liu, H. Li, J. Liu, Y. Wang, S. Tan, J. Yang, Y. Chen, M. Gruteser, and R. Martin. Leveraging wearables for steering and driver tracking. In Proceedings of IEEE INFOCOM, 2016. [3] S. Jain, C. Borgiattino, Y. Ren, M. Gruteser, Y. Chen, and C. Fabiana Chiasserini. Lookup: Enabling pedestrian safety services via shoe sensing. In Proceedings of ACM MobiSys, 2015. [4] S. Butterworth. On the theory of filter amplifiers. Wireless Engineer, 7(6):536–541, 1930. [5] M. R. Spiegel, S. Lipschutz, and D. Spellman. Vector analysis. Schaums Outlines (2nd ed.), pages 15–25, 2009. [6] K. Shoemake. Animating rotation with quaternion curves. Computer Graphics, 19(3):245–254, 1985. [7] G. Navarro. A guided tour to approximate string matching. ACM Computing Surveys, 33(11):31–88, 2001. [8] S. Consolvo, D. W. McDonald, T. Toscos, M. Y. Chen, J. Froehlich, B. Harrison, P. Klasnja, A. LaMarca, L. LeGrand, and R. Libby. Activity sensing in the wild: A field trial of ubifit garden. In Proceedings of ACM CHI, 2008. [9] K.-H. Chang, M. Y. Chen, and J. Canny. Tracking free-weight exercises. In Proceedings of ACM UbiComp, 2007. [10] A. Khany, S. Mellory, E. BerlinN, R. Thompsony, R. McNaneyy, P. Oliviery, and T. Plotzy. Beyond activity recognition: Skill assessment from accelerometer data. In Proceedings of ACM UbiComp, 2015. [11] N. Roy, H. Wang, and R. R. Choudhury. I am a smartphone and I can tell my user’s walking direction. In Proceedings of MobiSys, 2014. [12] P. Robertson, M. Angermann, and B. Krach. Simultaneous localization and mapping for pedestrians using only foot-mounted inertial sensors. In Proceedings of ACM UbiComp, 2009. [13] O. Woodman and R. Harle. Pedestrian localization for indoor environments. In Proceedings of ACM UbiComp, 2008. [14] H. Ding, L. Shangguan, Z. Yang, J. Han, Z. Zhou, P. Yang, W. Xi, and J. Zhao. Femo: A platform for free-weight exercise monitoring with rfids. In Proceedings of ACM SenSys, 2015. [15] W. Wang, A. X. Liu, M. Shahzad, K. Ling, and S. Lu. Understanding and modeling of wifi signal based human activity recognition. In Proceedings of ACM MobiCom, 2015. [16] Y. Wang, J. Liu, Y. Chen, M. Gruteser, J. Yang, and H. Liu. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proceedings of ACM MOBICOM, 2014. [17] J. Wang, D. Vasisht, and D. Katabi. RF-IDraw: Virtual touch screen in the air using RF signals. In Proc. of ACM SIGCOMM, 2014. [18] X. Zheng, J. Wang, L. Shangguan, Z. Zhou, and Y. Liu. Smokey: Ubiquitous smoking detection with commercial wifi infrastructures. In Proceedings of IEEE INFOCOM, 2016. [19] Q. Pu, S. Gupta, S. Gollakota, and S. Patel. Whole-home gesture recognition using wireless signals. In Proceedings of ACM MOBICOM, 2013