2146 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 Synchronize Inertial Readings From Multiple Mobile Devices in Spatial Dimension Lei Xie,Member,IEEE,Qingliang Cai,Student Member,IEEE,Alex X.Liu,Senior Member,IEEE, Wei Wang,Member;IEEE,Yafeng Yin,Member,IEEE,and Sanglu Lu,Member,IEEE Abstract-In this paper,we investigate the problem of space synchronization,i.e,synchronizing inertial readings from multiple mobile devices in the spatial dimension,in other words,mul- tiple mobile devices are space synchronized to have the same 3-D coordinates except that each device is the origin of its corresponding coordinate.We propose a scheme called MObile Space Synchronization (MOSS)for devices with two sensors: an accelerometer and a gyroscope,which are available on most Local Coordinat (a) mobile devices.Accelerometer readings from multiple mobile 6) devices on a human subject are used to achieve space synchroniza- Fig.1.Synchronize inertial readings from multi-devices in spatial dimension tion when the human subject is moving forward,such as walking (a)Synchronize 3-D coordinates among devices in VR games.(b)Local and running.Gyroscope readings from multiple mobile devices coordinates vs synchronized coordinates. on a human subject are used to maintain space synchronization when the human subject stops moving forward,which means that multiple mobile devices are space synchronized to have the we can no longer obtain the consistent acceleration caused by same 3-D coordinates except that each device is the origin body moving forward.Experiment results show that our MOSS scheme can achieve an average angle deviation of 9.8 and an of its corresponding coordinate.A modern mobile device is average measurement similarity of 97%. often equipped with an Inertial Measurement Unit (IMU), Index Terms-Space synchronization,mobile device which typically includes an accelerometer and a gyroscope, and sometimes a magnetometer as well for advanced models. I.INTRODUCTION A.Motivation The readings from IMU sensors are based on a local coordinate determined by the orientation of its body frame (or its mother TOWADAYS,the mobile devices equipped with inertial board to be more precise).Fig.1(b)shows the two local sensors are widely used to perform motion sensing and coordinates of a smartphone and a smartwatch,respectively, human computer interaction [1]-[4].For example,in virtual in solid arrows.For the multiple mobile devices placed at dif- reality games,a moving human subject may be equipped with ferent body parts of a human subject,due to the different orien- multiple mobile devices on different body parts(such as heads, tation of their body frames,their local coordinates are mostly arms,and legs)to capture the human's movement,as shown likely different.Without space synchronization,the readings in Fig.1(a).In this case,it is very essential to fuse the from the IMU sensors of different mobile devices are difficult readings from multiple mobile devices based on synchronized to be correlated with each other.With space synchronization, coordinates,so that they can be combined together to recover these readings can be used jointly to better describe human better orientation of the users'body parts and recognize the movements.We call the synchronized coordinate of a device as human activities. its global coordinate.Fig.1(a)and (b)shows the synchronized In this paper,we investigate the problem of space synchro- coordinates of the devices in dashed arrows.Space synchro- nization,i.e.,synchronizing inertial readings from multiple nization allows us to capture the movement of different body mobile devices in the spatial dimension,in other words, parts in the aligned 3-D coordinates,which helps to build more Manuscript received October 29,2017:revised May 10,2018;accepted accurate 3-D models of human actions. July 17,2018:approved by IEEE/ACM TRANSACTIONS ON NETWORKING A straightforward solution to mobile device space syn- Editor K.Tang.Date of publication August 14,2018;date of current chronization is to use compasses or magnetometer sensors in version October 15,2018.This work was supported in part by the National Natural Science Foundation of China under Grant 61472185,Grant 61472184 mobile devices.Magnetometer readings allow each device to Grant 61321491,Grant 61502224,and Grant 61702257,in part by the obtain the same magnetic field direction of the earth.Together Jiangsu Natural Science Foundation under Grant BK20151390 and Grant with the same gravity direction of each device,all devices BK20170648.in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035.in part by the National Science can therefore achieve space synchronization.This solution, Foundation under Grant CNS-1421407,in part by the Jiangsu Innovation and although simple,have two weaknesses.First,most state-of- Entrepreneurship (Shuangchuang)Program,and in part by the Collaborative the-art wearable devices (such as Apple Watch S1,Samsung Innovation Center of Novel Software Technology and Industrialization.(Cor responding authors:Alex X.Liu:Sanglu Lu.) Watch Gear 2,and MOTO Watch 360)are not equipped The authors are with the State Key Laboratory for Novel Software with magnetometer sensors;in comparison,accelerometer and Technology.Nanjing University,Nanjing 210023,China (e-mail:Ixie@nju. gyroscope sensors are available in most wearable devices.Sec- edu.cn:qingliangcai@dislab.nju.edu.cn:alexliu@nju.edu.cn:ww@nju.edu.cn: yafeng@nju.edu.cn:sanglu@nju.edu.cn). ond,magnetometer readings are often notoriously inaccurate Digital Object Identifier 10.1109/TNET.2018.2859246 in indoor environments 5,due to the magnetic interferences 1063-66922018 IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information
2146 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 Synchronize Inertial Readings From Multiple Mobile Devices in Spatial Dimension Lei Xie , Member, IEEE, Qingliang Cai, Student Member, IEEE, Alex X. Liu, Senior Member, IEEE, Wei Wang, Member, IEEE, Yafeng Yin, Member, IEEE, and Sanglu Lu, Member, IEEE Abstract— In this paper, we investigate the problem of space synchronization, i.e., synchronizing inertial readings from multiple mobile devices in the spatial dimension, in other words, multiple mobile devices are space synchronized to have the same 3-D coordinates except that each device is the origin of its corresponding coordinate. We propose a scheme called MObile Space Synchronization (MOSS) for devices with two sensors: an accelerometer and a gyroscope, which are available on most mobile devices. Accelerometer readings from multiple mobile devices on a human subject are used to achieve space synchronization when the human subject is moving forward, such as walking and running. Gyroscope readings from multiple mobile devices on a human subject are used to maintain space synchronization when the human subject stops moving forward, which means that we can no longer obtain the consistent acceleration caused by body moving forward. Experiment results show that our MOSS scheme can achieve an average angle deviation of 9.8◦ and an average measurement similarity of 97%. Index Terms— Space synchronization, mobile device. I. INTRODUCTION A. Motivation NOWADAYS, the mobile devices equipped with inertial sensors are widely used to perform motion sensing and human computer interaction [1]–[4]. For example, in virtual reality games, a moving human subject may be equipped with multiple mobile devices on different body parts (such as heads, arms, and legs) to capture the human’s movement, as shown in Fig. 1(a) . In this case, it is very essential to fuse the readings from multiple mobile devices based on synchronized coordinates, so that they can be combined together to recover better orientation of the users’ body parts and recognize the human activities. In this paper, we investigate the problem of space synchronization, i.e., synchronizing inertial readings from multiple mobile devices in the spatial dimension, in other words, Manuscript received October 29, 2017; revised May 10, 2018; accepted July 17, 2018; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor K. Tang. Date of publication August 14, 2018; date of current version October 15, 2018. This work was supported in part by the National Natural Science Foundation of China under Grant 61472185, Grant 61472184, Grant 61321491, Grant 61502224, and Grant 61702257, in part by the Jiangsu Natural Science Foundation under Grant BK20151390 and Grant BK20170648, in part by the Fundamental Research Funds for the Central Universities under Grant 020214380035, in part by the National Science Foundation under Grant CNS-1421407, in part by the Jiangsu Innovation and Entrepreneurship (Shuangchuang) Program, and in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization. (Corresponding authors: Alex X. Liu; Sanglu Lu.) The authors are with the State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210023, China (e-mail: lxie@nju. edu.cn; qingliangcai@dislab.nju.edu.cn; alexliu@nju.edu.cn; ww@nju.edu.cn; yafeng@nju.edu.cn; sanglu@nju.edu.cn). Digital Object Identifier 10.1109/TNET.2018.2859246 Fig. 1. Synchronize inertial readings from multi-devices in spatial dimension. (a) Synchronize 3-D coordinates among devices in VR games. (b) Local coordinates vs synchronized coordinates. multiple mobile devices are space synchronized to have the same 3-D coordinates except that each device is the origin of its corresponding coordinate. A modern mobile device is often equipped with an Inertial Measurement Unit (IMU), which typically includes an accelerometer and a gyroscope, and sometimes a magnetometer as well for advanced models. The readings from IMU sensors are based on a local coordinate determined by the orientation of its body frame (or its mother board to be more precise). Fig. 1(b) shows the two local coordinates of a smartphone and a smartwatch, respectively, in solid arrows. For the multiple mobile devices placed at different body parts of a human subject, due to the different orientation of their body frames, their local coordinates are mostly likely different. Without space synchronization, the readings from the IMU sensors of different mobile devices are difficult to be correlated with each other. With space synchronization, these readings can be used jointly to better describe human movements. We call the synchronized coordinate of a device as its global coordinate. Fig. 1(a) and (b) shows the synchronized coordinates of the devices in dashed arrows. Space synchronization allows us to capture the movement of different body parts in the aligned 3-D coordinates, which helps to build more accurate 3-D models of human actions. A straightforward solution to mobile device space synchronization is to use compasses or magnetometer sensors in mobile devices. Magnetometer readings allow each device to obtain the same magnetic field direction of the earth. Together with the same gravity direction of each device, all devices can therefore achieve space synchronization. This solution, although simple, have two weaknesses. First, most state-ofthe-art wearable devices (such as Apple Watch S1, Samsung Watch Gear 2, and MOTO Watch 360) are not equipped with magnetometer sensors; in comparison, accelerometer and gyroscope sensors are available in most wearable devices. Second, magnetometer readings are often notoriously inaccurate in indoor environments [5], due to the magnetic interferences 1063-6692 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2147 caused by the massive steel embedded in building concrete to these three axes.we can build the synchronized coordinates structures and other metallic objects.Our empirical study with each device as the origins. further shows that the magnetometer readings are not reli- Furthermore,we use gyroscope readings from multiple able in typical indoor environment.Specifically,for differ- mobile devices on a human subject to maintain space syn- ent positions in the indoor environment,the angle deviation chronization when the human subject stops moving forward, between the magnetometer measurement and the ground-truth which means that we can no longer obtain the consistent north can be as large as 110,the standard deviation of the acceleration caused by forward body motion.After the human magnetometer readings can be as large as 26.7;for a fixed subject stops moving forward,his body parts may still slightly position in the indoor environment,with the interferences from move or rotate.The gyroscope readings of a mobile device different sources such as smart phone,earphone,metal plate, allow us to continuously track the small rotations along and magnet,the angle deviation between the magnetometer the three axes of the device's local coordinate.We derive measurement and the ground-truth north can be as large as a real-time rotation matrix corresponding to the orientation 133.6,the standard deviation of the angle deviation can be variation by integrating the rotation rates in different axes over as large as 86.3 time.Leveraging the stability of the gravity direction in the synchronized coordinates,we further calibrate the estimated B.Proposed Approach rotation matrix from gyroscope tracking.Thus,we are able to In this paper,we propose a scheme called MObile Space maintain space synchronization even after the human subject Synchronization (MOSS)for devices with two sensors:an stops moving. accelerometer and a gyroscope,which are available on most mobile devices.Accelerometer readings from multiple mobile C.Technical Challenges and Solutions devices on a human subject are used to achieve space synchro- The first challenge is to extract the consistent acceleration nization when the human subject is moving forward,such as of body movement when the human subject moves forward. walking and running.Our insight on using accelerometer sen- This is challenging because both consistent and inconsistent sors to achieve space synchronization is that when the human accelerations are mixed together.As the mobile devices are subject moves forward,all attached mobile devices experience attached to different body locations of the human subject,they the same acceleration along the moving direction of the torso, perceive rather different accelerations in both the direction and which we call consistent acceleration.Although these mobile magnitude during human motion.To address this challenge, devices also experience various other accelerations due to we propose a principal component analysis (PCA)based intra-body movements (such as arm and leg movements), scheme to remove the inconsistent accelerations from the which we call inconsistent acceleration.they are usually much observed accelerations.The key observation is that the con- smaller than the consistent acceleration because intra-body sistent acceleration contributes to the observed accelerations movements are usually orders of magnitude smaller than the of all mobile devices attached to the human subject.In other forward movement.When the human subject is moving for- words,the observed acceleration signal of each device is the ward,the approach of simply adding an inertial sensor on the combination of the consistent acceleration signal,which is human chest to directly measure the consistent acceleration from the forwarding movement and the same for all devices, seems to be a feasible solution.however,it cannot work and the inconsistent acceleration signal,which is from the effectively due to the following reasons:First,it still requires intra-body movement from and unique to the device itself. the other mobile devices to synchronize with this specified Furthermore,we observe that the inconsistent acceleration device in the spatial dimension,as the consistent acceleration from the intra-body movement cancels each other out during can only be extracted directly from this device rather than the back and forth movement,and its expected value is close all devices.Usually a magnetometer is essentially required to 0 within a large enough time interval.Thus,the observed for all devices to achieve this synchronization,however,this acceleration signals of multiple devices are strongly correlated, is contradictory to the situation we need to tackle.Second. and therefore we can use PCA-based approach to cancel out even for this specified device.it may lead to inaccuracy the inconsistent factor and extract the consistent factor. in directly measuring the consistent acceleration,since the The second challenge is to address the accumulated errors moving human body may usually introduce some inconsistent in maintaining space synchronization when the human sub- accelerations,more or less,into the measurements on the hor- ject stops moving forward.It is well known that the errors izontal plane.Hence,in this paper,we choose to compute the in gyroscope based oriental tracking accumulate [6],[7]. forwarding direction by extracting the consistent acceleration Furthermore,the errors are further exacerbated by the large from the accelerometer readings.We can treat the inconsistent angular velocities and linear accelerations in human body acceleration as noises and use signal processing techniques motion.Current solutions primarily rely on Kalman filters; to filter them out mostly.Moreover,using a low pass filter however,they only use a single data source from the gyroscope such as a Butterworth filter,the gravitational acceleration to calibrate the accumulated errors [8],which is not sufficient can be extracted from the acceleration measurements in the to further reduce the errors in maintaining space synchro- local coordinate system.Therefore,based on the forwarding nization.To address this challenge,we propose a simple direction and the gravitational direction as reference axes, but effective complementary filter to calibrate the gyroscope the third axis can also be obtained since it is perpendicular to tracking based on the stability of the gravity direction.We the plane defined by these two reference axes.Thus,according propose a rotation model that defines the rotations in the
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2147 caused by the massive steel embedded in building concrete structures and other metallic objects. Our empirical study further shows that the magnetometer readings are not reliable in typical indoor environment. Specifically, for different positions in the indoor environment, the angle deviation between the magnetometer measurement and the ground-truth north can be as large as 110◦, the standard deviation of the magnetometer readings can be as large as 26.7◦; for a fixed position in the indoor environment, with the interferences from different sources such as smart phone, earphone, metal plate, and magnet, the angle deviation between the magnetometer measurement and the ground-truth north can be as large as 133.6◦, the standard deviation of the angle deviation can be as large as 86.3◦. B. Proposed Approach In this paper, we propose a scheme called MObile Space Synchronization (MOSS) for devices with two sensors: an accelerometer and a gyroscope, which are available on most mobile devices. Accelerometer readings from multiple mobile devices on a human subject are used to achieve space synchronization when the human subject is moving forward, such as walking and running. Our insight on using accelerometer sensors to achieve space synchronization is that when the human subject moves forward, all attached mobile devices experience the same acceleration along the moving direction of the torso, which we call consistent acceleration. Although these mobile devices also experience various other accelerations due to intra-body movements (such as arm and leg movements), which we call inconsistent acceleration, they are usually much smaller than the consistent acceleration because intra-body movements are usually orders of magnitude smaller than the forward movement. When the human subject is moving forward, the approach of simply adding an inertial sensor on the human chest to directly measure the consistent acceleration seems to be a feasible solution, however, it cannot work effectively due to the following reasons: First, it still requires the other mobile devices to synchronize with this specified device in the spatial dimension, as the consistent acceleration can only be extracted directly from this device rather than all devices. Usually a magnetometer is essentially required for all devices to achieve this synchronization, however, this is contradictory to the situation we need to tackle. Second, even for this specified device, it may lead to inaccuracy in directly measuring the consistent acceleration, since the moving human body may usually introduce some inconsistent accelerations, more or less, into the measurements on the horizontal plane. Hence, in this paper, we choose to compute the forwarding direction by extracting the consistent acceleration from the accelerometer readings. We can treat the inconsistent acceleration as noises and use signal processing techniques to filter them out mostly. Moreover, using a low pass filter such as a Butterworth filter, the gravitational acceleration can be extracted from the acceleration measurements in the local coordinate system. Therefore, based on the forwarding direction and the gravitational direction as reference axes, the third axis can also be obtained since it is perpendicular to the plane defined by these two reference axes. Thus, according to these three axes, we can build the synchronized coordinates with each device as the origins. Furthermore, we use gyroscope readings from multiple mobile devices on a human subject to maintain space synchronization when the human subject stops moving forward, which means that we can no longer obtain the consistent acceleration caused by forward body motion. After the human subject stops moving forward, his body parts may still slightly move or rotate. The gyroscope readings of a mobile device allow us to continuously track the small rotations along the three axes of the device’s local coordinate. We derive a real-time rotation matrix corresponding to the orientation variation by integrating the rotation rates in different axes over time. Leveraging the stability of the gravity direction in the synchronized coordinates, we further calibrate the estimated rotation matrix from gyroscope tracking. Thus, we are able to maintain space synchronization even after the human subject stops moving. C. Technical Challenges and Solutions The first challenge is to extract the consistent acceleration of body movement when the human subject moves forward. This is challenging because both consistent and inconsistent accelerations are mixed together. As the mobile devices are attached to different body locations of the human subject, they perceive rather different accelerations in both the direction and magnitude during human motion. To address this challenge, we propose a principal component analysis (PCA) based scheme to remove the inconsistent accelerations from the observed accelerations. The key observation is that the consistent acceleration contributes to the observed accelerations of all mobile devices attached to the human subject. In other words, the observed acceleration signal of each device is the combination of the consistent acceleration signal, which is from the forwarding movement and the same for all devices, and the inconsistent acceleration signal, which is from the intra-body movement from and unique to the device itself. Furthermore, we observe that the inconsistent acceleration from the intra-body movement cancels each other out during the back and forth movement, and its expected value is close to 0 within a large enough time interval. Thus, the observed acceleration signals of multiple devices are strongly correlated, and therefore we can use PCA-based approach to cancel out the inconsistent factor and extract the consistent factor. The second challenge is to address the accumulated errors in maintaining space synchronization when the human subject stops moving forward. It is well known that the errors in gyroscope based oriental tracking accumulate [6], [7]. Furthermore, the errors are further exacerbated by the large angular velocities and linear accelerations in human body motion. Current solutions primarily rely on Kalman filters; however, they only use a single data source from the gyroscope to calibrate the accumulated errors [8], which is not sufficient to further reduce the errors in maintaining space synchronization. To address this challenge, we propose a simple but effective complementary filter to calibrate the gyroscope tracking based on the stability of the gravity direction. We propose a rotation model that defines the rotations in the
2148 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 orthogonal and parallel directions of the gravity,respectively. We then use this model to calibrate the gyroscope-based estimation with the Minimum Mean Square Error (MMSE) estimator. Smant Phone:斜Sumoung3 D.Summary of Experimental Results We implemented our MOSS system on mobile devices including Google Glass and Samsung S5 smart phones.They Fig.2.Measurement setup are deployed at different body locations of each human subject. We let the human subjects walk along different types of traces angles of a human subject so that the indoor pathway maps in outdoor environments with different moving modes,i.e., can be obtained [21].Wang et al.[22]used compasss,gyro- walk,run,and jump.We use two main metrics to evaluate scopes,and WiFi landmarks to estimate the absolute walking the performance:(1)angle accuracy:the angle deviation direction of a human subject.Of prior work in this category, between the estimated human body movement direction and only the APT system does not use magnetometer sensors [24]; the ground truth,and (2)coordinate accuracy:the similarity instead,APT uses accelerometer and gyroscope sensors to between the synchronized coordinate and the ground truth. obtain the walking direction of a human subject.Compared Experiment results show that MOSS achieves an average angle with our work,both the WalkCompass and APT does not accuracy of 9.8 and an average coordinate accuracy of 97%. address space synchronization among multiple mobile devices A real-world case study with free activities further shows that as they use only one smartphone.In contrast,in this paper MOSS achieves an average angle accuracy of 12 and an we investigate multiple mobile devices instead of one device average coordinate accuracy of 91%. to synchronize the inertial readings from multiple devices in spatial dimension.We propose a more generalized solution II.RELATED WORK to heading direction estimation for multiple wearable devices Orientation Estimation:Much work has been done on subject to different but correlated accelerations.Our solution estimating the orientation of a mobile device (such as a neither relies on the inaccurate magnetometer measurements smartphone)using accelerometer,gyroscope,and magne- nor uses any application specific features tometer sensors [2],[6].[9]-[17].Compared with our work, most of such work uses magnetometers.For example III.UNDERSTANDING HUMAN MOTIONS Madgwick et al.[13]proposed a quaternion representation to incorporate accelerometer and magnetometer readings for A.Measurements orientation estimation.Zhou et al.[6]used the accelerometer Measurement Setup:We placed six mobile devices,one and magnetometer to assist gyroscope in orientation estimation smart glass(Google Glass 2)and five smartphones(Samsung by selecting the best sensing capabilities.Gowda et al.[14] Galaxy S5).at different body location of a human subject tried to map from a local frame of the sensor to a global frame to continuously collect the inertial measurements in his daily of reference,so as to track a ball's 3D trajectory and spin life.As shown in Fig.2,the Google Glass was placed on with inertial sensors and radios embedded in the ball.Of prior the head and the five phones were placed at five different work in this category,only two systems,Autowitness [18]locations on the body.These devices are all equipped with an and Nericell [19],which finds the rotation matrix between the accelerometer,a gyroscope,and a magnetometer.We use the local coordinate of a smartphone and the reference coordinate three axes obtained from the magnetometers as the reference by the acceleration measurement,did not use magnetometers. global coordinate. They used a vehicle's forwarding accelerations (i.e.,speeding Measurement of Human Body Movements:We observed that up and slowing down)obtained from accelerometer readings. when the human subject attached with multiple mobile devices Compared with our work,they have two key limitations as moves forward,the inconsistent accelerations from intra-body they attach mobile devices to moving vehicles rather than movement have different directions and magnitude.To extract human subjects.First,as there is no intra-body movements the consistent and the inconsistent accelerations,we let the and a vehicle moves much straighter than human moves,they human subject walk along a straight path for 30 seconds.We do not need to deal with our first technical challenge.Second, attached an additional IMU sensor on the chest to estimate the as the mobile devices do not move after a vehicle stops,they ground-truth of the consistent acceleration as the inconsistent do not need to deal with our second technical challenge. accelerations on the chest is negligible.We collected the Direction Estimation:Recently some work explored the acceleration measurements from the six devices as mixed estimation of heading directions for dead reckoning-based accelerations and then extract the inconsistent accelerations navigation schemes [20]-[26].Compared with our work,most from each device by subtracting the consistent acceleration. of such work uses magnetometers to estimate the heading To illustrate the direction and magnitude of both the consistent directions while trying to mitigate the magnetic interference and inconsistent accelerations,we plot them as vectors in the from indoor environments.For example,WalkCompass used polar coordinate system corresponding to the earth coordinate magnetometers to estimate the walking direction of a human system.We plot a vector of the inconsistent acceleration subject [20.Walkie-Markie used the magnetometer and gyro- for every 100 ms during the time interval of 5 seconds. scopes in smartphones to get the walking direction and turning Fig.3 shows the directions and magnitudes of the original
2148 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 orthogonal and parallel directions of the gravity, respectively. We then use this model to calibrate the gyroscope-based estimation with the Minimum Mean Square Error (MMSE) estimator. D. Summary of Experimental Results We implemented our MOSS system on mobile devices including Google Glass and Samsung S5 smart phones. They are deployed at different body locations of each human subject. We let the human subjects walk along different types of traces in outdoor environments with different moving modes, i.e., walk, run, and jump. We use two main metrics to evaluate the performance: (1) angle accuracy: the angle deviation between the estimated human body movement direction and the ground truth, and (2) coordinate accuracy: the similarity between the synchronized coordinate and the ground truth. Experiment results show that MOSS achieves an average angle accuracy of 9.8◦ and an average coordinate accuracy of 97%. A real-world case study with free activities further shows that MOSS achieves an average angle accuracy of 12◦ and an average coordinate accuracy of 91%. II. RELATED WORK Orientation Estimation: Much work has been done on estimating the orientation of a mobile device (such as a smartphone) using accelerometer, gyroscope, and magnetometer sensors [2], [6], [9]–[17]. Compared with our work, most of such work uses magnetometers. For example, Madgwick et al. [13] proposed a quaternion representation to incorporate accelerometer and magnetometer readings for orientation estimation . Zhou et al. [6] used the accelerometer and magnetometer to assist gyroscope in orientation estimation by selecting the best sensing capabilities. Gowda et al. [14] tried to map from a local frame of the sensor to a global frame of reference, so as to track a ball’s 3D trajectory and spin with inertial sensors and radios embedded in the ball. Of prior work in this category, only two systems, Autowitness [18] and Nericell [19], which finds the rotation matrix between the local coordinate of a smartphone and the reference coordinate by the acceleration measurement, did not use magnetometers. They used a vehicle’s forwarding accelerations (i.e., speeding up and slowing down) obtained from accelerometer readings. Compared with our work, they have two key limitations as they attach mobile devices to moving vehicles rather than human subjects. First, as there is no intra-body movements and a vehicle moves much straighter than human moves, they do not need to deal with our first technical challenge. Second, as the mobile devices do not move after a vehicle stops, they do not need to deal with our second technical challenge. Direction Estimation: Recently some work explored the estimation of heading directions for dead reckoning-based navigation schemes [20]–[26]. Compared with our work, most of such work uses magnetometers to estimate the heading directions while trying to mitigate the magnetic interference from indoor environments. For example, WalkCompass used magnetometers to estimate the walking direction of a human subject [20]. Walkie-Markie used the magnetometer and gyroscopes in smartphones to get the walking direction and turning Fig. 2. Measurement setup. angles of a human subject so that the indoor pathway maps can be obtained [21]. Wang et al. [22] used compasss, gyroscopes, and WiFi landmarks to estimate the absolute walking direction of a human subject. Of prior work in this category, only the APT system does not use magnetometer sensors [24]; instead, APT uses accelerometer and gyroscope sensors to obtain the walking direction of a human subject. Compared with our work, both the WalkCompass and APT does not address space synchronization among multiple mobile devices as they use only one smartphone. In contrast, in this paper we investigate multiple mobile devices instead of one device to synchronize the inertial readings from multiple devices in spatial dimension. We propose a more generalized solution to heading direction estimation for multiple wearable devices subject to different but correlated accelerations. Our solution neither relies on the inaccurate magnetometer measurements nor uses any application specific features. III. UNDERSTANDING HUMAN MOTIONS A. Measurements Measurement Setup: We placed six mobile devices, one smart glass (Google Glass 2) and five smartphones (Samsung Galaxy S5), at different body location of a human subject to continuously collect the inertial measurements in his daily life. As shown in Fig. 2, the Google Glass was placed on the head and the five phones were placed at five different locations on the body. These devices are all equipped with an accelerometer, a gyroscope, and a magnetometer. We use the three axes obtained from the magnetometers as the reference global coordinate. Measurement of Human Body Movements: We observed that when the human subject attached with multiple mobile devices moves forward, the inconsistent accelerations from intra-body movement have different directions and magnitude. To extract the consistent and the inconsistent accelerations, we let the human subject walk along a straight path for 30 seconds. We attached an additional IMU sensor on the chest to estimate the ground-truth of the consistent acceleration as the inconsistent accelerations on the chest is negligible. We collected the acceleration measurements from the six devices as mixed accelerations and then extract the inconsistent accelerations from each device by subtracting the consistent acceleration. To illustrate the direction and magnitude of both the consistent and inconsistent accelerations, we plot them as vectors in the polar coordinate system corresponding to the earth coordinate system. We plot a vector of the inconsistent acceleration for every 100 ms during the time interval of 5 seconds. Fig. 3 shows the directions and magnitudes of the original
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2149 12090180 120 9010 120 90180 120 902%0 150 30 150 30 150 30 150 10 30 180 180 0 180 180 210 330 210 330 210 330 210 330 240270300 240270300 240270300 240270300 (a b (c) (d) Fig.3.The directions (and magnitudes (m/s2)of the original/consistent/inconsistent accelerations in different devices.(a)S1.(b)S2.(c)S3.(d)56. ■出口多塞法口型 口出强口澳类多口类 口光口器口数图 (2) Fig.4.The mean and standard deviation of different accelerations.(a)The mixed accelerations (m/s2).(b)The consistent accelerations (m/s?).(c)The inconsistent accelerations (m/s). accelerations,the consistent accelerations,and the inconsistent acceleration f(t)and inconsistent acceleration f(t),ie., accelerations,respectively,with the color green,red and blue. fi(t)=fe(t)+f(t).The accelerations are originally measured Without loss of generality,we plot these accelerations in the according to the local coordinate system of the device.As polar system for the devices S1,S2,S3 and S6.We observed the human subject is moving,the local coordinate system that,for the inconsistent accelerations,the average magnitudes is rotating over time relative to the earth coordinate system; of the inconsistent accelerations are different among different thus.the acceleration measurements cannot describe human devices,because they depend on the motion amplitudes in motions consistently.Suppose we can build a fixed coordinate different body parts.Moreover,the directions of the incon- system relative to the earth coordinate system.For the fixed sistent accelerations are distributed by and large evenly on coordinate system,we use fi(t),fi.(t),and fi.=(t)to denote the horizontal plane,because the accelerations of intra-body the projections of fi(t)in y and 2 axes,respectively,and movements often mutually offset each other in opposite direc- use f(t),f(t).and f (t)to denote the projections of tions due to back and forth body part movement such as arm f(t)in y.and z axes,respectively.Let the angles between swing. fe(t)and each x,y and z axis at time t be ai(t).Bi(t). We observed that the mean value of inconsistent accel- and i(t),respectively.Since the consistent acceleration fe(t) erations is close to 0 in each direction for a sufficiently has a fixed direction with respect to the coordinate system, large time window (such as 10 seconds),although the stan- i(t),Bi(t),and (t)are all constant values over time;thus, dard deviations of inconsistent accelerations are fairly large. we denote them as ai,Bi,and i,respectively,for simplicity. Fig.4 shows the mean and standard deviation of the mixed, Thus,the direction of fe(t)can be represented as a unit vector consistent,and inconsistent accelerations,respectively,in the (cos ai,cos Bi,cosYi).Let fe(t)denote the magnitude of the forward-back direction and the left-right direction of human consistent acceleration in the forwarding direction;thus,its motions.We observed that although the standard deviations projection on the y and z axes are fe(t)cos ai,fe(t)cos Bi, of all accelerations are fairly large,their mean values are all and fe(t)cosi,respectively.The projections on the three axes relatively small.In regard to the inconsistent accelerations, are as follows: for both directions.the standard deviations are usually in the order of 1m/s2.The standard deviations in the forward-back fi.r(t)=fe(t)cosai+fir(t), direction are significantly greater than the left-right direction, fi.y(t)=fe(t)cosBi+fi.y(t), (1) due to the reason that the human subject is moving for- fi.z(t)=fe(t)cosvi+fz(t). ward.However.for both directions.the mean values of the inconsistent accelerations are all in the order of 10-5m/s2. Fig.5 shows the relationship among fi(t),fe(t),and f(t), This implies that,although the inconsistent accelerations have as well as the projection of fe(t)on the three axes. different directions and magnitudes,the overall impact of the Thus,if we can compute the value of cos ai,cos Bi,and inconsistent accelerations from the intra-body movement can cos i,we can calculate the direction of the consistent acceler- be negligible statistically. ation f(t)in the corresponding coordinate system.Further,for any inconsistent accelerations corresponding to the intra-body movement,we observe that the mean value of the inconsistent B.Modeling of Human Motions accelerations is close to 0,as long as the time interval Let fi(t)denote the mixed acceleration measured from is large enough.The following theorem captures the above device Di at time t.Here fi(t)consists of consistent characteristics:
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2149 Fig. 3. The directions (◦) and magnitudes (m/s2) of the original/consistent/inconsistent accelerations in different devices. (a) S1. (b) S2. (c) S3. (d) S6. Fig. 4. The mean and standard deviation of different accelerations. (a) The mixed accelerations (m/s2). (b) The consistent accelerations (m/s2). (c) The inconsistent accelerations (m/s2). accelerations, the consistent accelerations, and the inconsistent accelerations, respectively, with the color green, red and blue. Without loss of generality, we plot these accelerations in the polar system for the devices S1, S2, S3 and S6. We observed that, for the inconsistent accelerations, the average magnitudes of the inconsistent accelerations are different among different devices, because they depend on the motion amplitudes in different body parts. Moreover, the directions of the inconsistent accelerations are distributed by and large evenly on the horizontal plane, because the accelerations of intra-body movements often mutually offset each other in opposite directions due to back and forth body part movement such as arm swing. We observed that the mean value of inconsistent accelerations is close to 0 in each direction for a sufficiently large time window (such as 10 seconds), although the standard deviations of inconsistent accelerations are fairly large. Fig. 4 shows the mean and standard deviation of the mixed, consistent, and inconsistent accelerations, respectively, in the forward-back direction and the left-right direction of human motions. We observed that although the standard deviations of all accelerations are fairly large, their mean values are all relatively small. In regard to the inconsistent accelerations, for both directions, the standard deviations are usually in the order of 1m/s2. The standard deviations in the forward-back direction are significantly greater than the left-right direction, due to the reason that the human subject is moving forward. However, for both directions, the mean values of the inconsistent accelerations are all in the order of 10−5m/s2. This implies that, although the inconsistent accelerations have different directions and magnitudes, the overall impact of the inconsistent accelerations from the intra-body movement can be negligible statistically. B. Modeling of Human Motions Let fi(t) denote the mixed acceleration measured from device Di at time t. Here fi(t) consists of consistent acceleration fc(t) and inconsistent acceleration f i(t), i.e., fi(t) = fc(t)+f i(t). The accelerations are originally measured according to the local coordinate system of the device. As the human subject is moving, the local coordinate system is rotating over time relative to the earth coordinate system; thus, the acceleration measurements cannot describe human motions consistently. Suppose we can build a fixed coordinate system relative to the earth coordinate system. For the fixed coordinate system, we use fi,x(t), fi,y(t), and fi,z(t) to denote the projections of fi(t) in x, y and z axes, respectively, and use f i,x(t), f i,y(t), and f i,z(t) to denote the projections of f i(t) in x, y, and z axes, respectively. Let the angles between fc(t) and each x, y and z axis at time t be αi(t), βi(t), and γi(t), respectively. Since the consistent acceleration fc(t) has a fixed direction with respect to the coordinate system, αi(t), βi(t), and γi(t) are all constant values over time; thus, we denote them as αi, βi, and γi, respectively, for simplicity. Thus, the direction of fc(t) can be represented as a unit vector cos αi, cos βi, cos γi. Let fc(t) denote the magnitude of the consistent acceleration in the forwarding direction; thus, its projection on the x, y and z axes are fc(t) cos αi, fc(t) cos βi, and fc(t) cos γi, respectively. The projections on the three axes are as follows: ⎧ ⎪⎨ ⎪⎩ fi,x(t) = fc(t) cos αi + f i,x(t), fi,y(t) = fc(t) cos βi + f i,y(t), fi,z(t) = fc(t) cos γi + f i,z(t). (1) Fig.5 shows the relationship among fi(t), fc(t), and f i(t), as well as the projection of fc(t) on the three axes. Thus, if we can compute the value of cos αi, cos βi, and cos γi, we can calculate the direction of the consistent acceleration fc(t) in the corresponding coordinate system. Further, for any inconsistent accelerations corresponding to the intra-body movement, we observe that the mean value of the inconsistent accelerations is close to 0, as long as the time interval is large enough. The following theorem captures the above characteristics:
2150 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 Z axis "地 Z axis constant f System Bi Y axis 阳 gravitational Y axis acceleration g Fig.5.fi(t)consists of consistent acceleration f(t)and inconsistent acceleration f(t).the projection of fe(t)on the r.y and 2 axes are Fig.6.Deriving the global coordinate. fe(t)cos ai,fe(t)cos Bi,and fe(t)cos Yi.respectively. Theorem 1:For a fixed coordinate system relative to the A vector quantity v defined in the local coordinate system earth coordinate system,for any inconsistent acceleration f.(t) is equivalent to the vector v=C.v defined in the global caused by intra-body movements,the expected value of f(t) coordinate.The inverse transformation is v=C.v as the is approximately equal to 0 within a sufficiently large time inverse of a rotation matrix is equal to its transpose. interval [ts,te].That is, For the local coordinate of a device,assuming that we can extract a constant acceleration as a vector fe from the E':（训=te-t。J 1 E(t)dt≈0. (2) acceleration measurements and extract a constant gravitational acceleration as a vector g from the low pass filter (such as the Proof:Without loss of generality,we prove this theorem Butterworth filter [271).we can build the global coordinate and for z-axis.Our reasoning is applicable for other axes.Let compute the rotation matrix as follows.After we obtain the Avi.denote the change of the relative velocity between gravity vector g,we derive its opposite value and normalize device Di and the body after the intra-body movements this vector as we then set this vector to represent occurred during time interval [ts,te].According to the rela- the global Z-axis as it is in the opposite direction of the tion between velocity and acceleration,we have Avi.r= gravitational acceleration.We set the vector zo to represent f(t)dt.As body parts move back and forth,the ampli- to be perpendicular to the horizontal plane.After computing tudes of the intra-body movement are usually small and do the cross product y =g x fc,we obtain a vector y that is not change significantly in any direction.Thus,Avi.should perpendicular to the plane determined by the two distinct but be smaller than a constant threshold C.In general situations, intersecting lines corresponding to g and fe.We normalize this CC,then Ef(d≤%ms2=2× obtain the vectors xo,yo,and zo in the local coordinate,which 10-2m/s2≈0. correspond to the global X,Y,and Z-axes,we can derive the rotation matrix CT.Fig.6 illustrates the above process. Theorem 1 implies that,during a sufficiently large time interval,e.g.,10~20s,the expected value of the inconsistent acceleration can be negligible,as the inconsistent accelerations B.Maintain Space Synchronization cancel each other out during the back and forth movements As the orientation of a device may be continuously chang- This explains our observations that the mean value of the ing,we need to track and update the rotation matrix C over inconsistent acceleration is in the order of 10-5m/s2. time.For example,if the rotation matrix at time t is given by Ct,then the rotation matrix C+6t at time t+6t can be IV.SPACE SYNCHRONIZATION MODEL computed as the product of two matrices A.Achieve Space Synchronization C1+8t=CiAt.t+8t (4) The objective of space synchronization for mobile devices where At.t+6t is the rotation matrix relating the local coordi- is to align their local coordinate to their global coordinate nate at time t to the one at time t+ot.To estimate the matrix We use the direction cosine representation to quantify the orientation difference between the local and global coordi- At.t+6t,we use gyroscope measurements.According to the nates of each device.In the direction cosine representation, small angle approximation [12],if we use w(t),w(t),and the orientation of the local coordinate relative to the global w(t)to represent the rotation rate of small rotations about its coordinate system is specified by a 3 x 3 rotation matrix C. x,y and z axes between time t and t+ot,then where each column is a unit vector along one axis in the local rt+6t coordinate specified in terms of the global coordinate axes. At.t+6t exp( (t)dt) (5)
2150 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 Fig. 5. fi(t) consists of consistent acceleration fc(t) and inconsistent acceleration f i(t), the projection of fc(t) on the x, y and z axes are fc(t) cos αi, fc(t) cos βi, and fc(t) cos γi, respectively. Theorem 1: For a fixed coordinate system relative to the earth coordinate system, for any inconsistent acceleration f i(t) caused by intra-body movements, the expected value of f i(t) is approximately equal to 0 within a sufficiently large time interval [ts, te]. That is, E[f’i(t)] = 1 te − ts te ts f i(t)dt ≈ 0. (2) Proof: Without loss of generality, we prove this theorem for x-axis. Our reasoning is applicable for other axes. Let Δvi,x denote the change of the relative velocity between device Di and the body after the intra-body movements occurred during time interval [ts, te]. According to the relation between velocity and acceleration, we have Δvi,x = te ts f i,x(t)dt. As body parts move back and forth, the amplitudes of the intra-body movement are usually small and do not change significantly in any direction. Thus, Δvi,x should be smaller than a constant threshold C. In general situations, C ≤ 0.4m/s due to the back-and-forth moving property. Thus, let Δt = te − ts, for E[f i,x(t)], i.e., the expected value of the acceleration f i,x(t) during [ts, te], we have E[f i,x(t)] = 1 Δt te ts f i,x(t)dt = Δvi,x Δt < C Δt . (3) If the time interval Δt = te − ts is sufficiently larger than the value of C, then E[f i,x(t)] ≈ 0, e.g., Δt = 20s and C = 0.4m/s, Δt C, then E[f i,x(t)] ≤ 0.4 20 m/s2 = 2 × 10−2m/s2 ≈ 0. Theorem 1 implies that, during a sufficiently large time interval, e.g., 10∼20s, the expected value of the inconsistent acceleration can be negligible, as the inconsistent accelerations cancel each other out during the back and forth movements. This explains our observations that the mean value of the inconsistent acceleration is in the order of 10−5m/s2. IV. SPACE SYNCHRONIZATION MODEL A. Achieve Space Synchronization The objective of space synchronization for mobile devices is to align their local coordinate to their global coordinate. We use the direction cosine representation to quantify the orientation difference between the local and global coordinates of each device. In the direction cosine representation, the orientation of the local coordinate relative to the global coordinate system is specified by a 3 × 3 rotation matrix C, where each column is a unit vector along one axis in the local coordinate specified in terms of the global coordinate axes. Fig. 6. Deriving the global coordinate. A vector quantity vl defined in the local coordinate system is equivalent to the vector vg = C · vl defined in the global coordinate. The inverse transformation is vl = CT · vg as the inverse of a rotation matrix is equal to its transpose. For the local coordinate of a device, assuming that we can extract a constant acceleration as a vector fc from the acceleration measurements and extract a constant gravitational acceleration as a vector g from the low pass filter (such as the Butterworth filter [27]), we can build the global coordinate and compute the rotation matrix as follows. After we obtain the gravity vector g, we derive its opposite value and normalize this vector as zo = −g g , we then set this vector to represent the global Z-axis as it is in the opposite direction of the gravitational acceleration. We set the vector zo to represent to be perpendicular to the horizontal plane. After computing the cross product y = g × fc, we obtain a vector y that is perpendicular to the plane determined by the two distinct but intersecting lines corresponding to g and fc. We normalize this vector as yo = y y . Since the vector yo is on the horizontal plane, we set this vector to represent the global Y -axis. After that, by computing the cross product x = g × y, we obtain a vector x that is orthogonal to the plane determined by the two distinct but intersecting lines corresponding to g and y. We normalize this vector as xo = x x . Since the vector xo is on the horizontal plane, and it is orthogonal to the global Y -axis and Z-axis, we set it to represent the global X-axis. After we obtain the vectors xo, yo, and zo in the local coordinate, which correspond to the global X, Y , and Z-axes, we can derive the rotation matrix CT . Fig. 6 illustrates the above process. B. Maintain Space Synchronization As the orientation of a device may be continuously changing, we need to track and update the rotation matrix C over time. For example, if the rotation matrix at time t is given by Ct, then the rotation matrix Ct+δt at time t + δt can be computed as the product of two matrices Ct+δt = CtAt,t+δt (4) where At,t+δt is the rotation matrix relating the local coordinate at time t to the one at time t+δt. To estimate the matrix At,t+δt, we use gyroscope measurements. According to the small angle approximation [12], if we use ωx(t), ωy(t), and ωz(t) to represent the rotation rate of small rotations about its x, y and z axes between time t and t + δt, then At,t+δt = exp( t+δt t Ω(t)dt) (5)
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2151 record the accelerometer and gyroscope measurements after receiving this beacon.Let this time be to. (2)Local Measurement with Gyroscope Tracking:For each time t within a time window W.each device D;extracts the gravitational acceleration g(t)and the linear acceleration fi(t) from the acceleration measurements in its local coordinate Meanwhile,each device D;tracks the rotation speed wi(t) in each axis from the gyroscope,and computes the rotation matrix Af.t from time toto t.After time window W,device Di sends the following to the server:the rotation matrix Fig.7.System architecture. Atthe gravity gi(t),and the linear acceleration fi(t)for all t∈W. where (3)PCA Analysis:After receiving the responses from all 0 -w2(t) wy(t) devices,the server performs PCA to extract the magnitude and 2(t)= wz(t) 0 -w:(t) (6 estimates the direction of the consistent acceleration f(t)in -wy(t) wz(t) 0 the local coordinate of each device.With the gravity direction, We further use Taylor expansion of the exponential term to it constructs a rotation matrix C for each device at time to. approximate the value of exp(() and sends C to each device. (4)Local Rotation Update:Each device Di locally updates As the angular velocity signals obtained from the gyro- the rotation matrix C using the matrix A.from gyroscope scopes are"integrated"to track and update the rotation matrix, tracking,i.e..C=CAIt then transforms the accelerom- the errors in the gyroscope signals propagate into the calcu- eter and gyroscope measurements from the local coordinate to lated orientation leading to large cumulative tracking errors. global coordinate. The tracking errors in the rotation matrix At.t+t are amplified in each axis through the exponential function in Eq.(5).The VI.CONSISTENT ACCELERATION EXTRACTION larger the angular velocities and linear accelerations in the forward moving direction are,the more the tracking errors As the accelerometer measurements from multi-devices are e(t).ey(t),and e=(t)in that direction.Thus,the rotation highly correlated,we perform Principal Component Analy- sis (PCA)to extract the consistent acceleration during the matrix At.t+t derived from gyroscope tracking is prone to process of moving forward.This consists of three steps: large accumulation of errors after long periods.It is essential preprocessing,principal component extraction,and direction to leverage other measurements (such as the accelerometer estimation. measurements)for further calibration. V.SYSTEM OVERVIEW A.Preprocessing The system architecture is shown in Fig.7.MOSS con- Suppose the acceleration measurements fi(t)for each device sists of two components:a Consistent Direction Estimator Di can be obtained as (xi(t),(t),zi(t))from the x,y,and for achieving space synchronization,and a Gyroscope-based z-axes in its local coordinate Li(t)at time t.Then,we can Orientation Tracker for maintaining space synchronization. use the rotation matrix(A)T to transform the acceleration Consistent Direction Estimator uses principal component measurements from the current local coordinate Li(t)at time analysis to extract the magnitude of consistent acceleration t to the reference local coordinate Li(to)at time to,i.e.. from multiple devices,and further estimate the direction of zi(to) x:(t) consistent acceleration.Combined with the gravity direction h(to） (Ai)T (t) (7) extracted from the acceleration measurements,it generates a 2:(to)】 2(t) rotation matrix for space synchronization.Gyroscope-based Orientation Tracker continuously estimates the orientation Thus,we obtain the acceleration measurements in the refer- variation of the body frame from the gyroscope measurements. ence local coordinate,i.e.,the local coordinate at time to, Leveraging the stability of the gravitational accelerations in the which is therefore consistent over time. global coordinate,we calibrate the rotation matrix using the Further,as we focus on extracting the forwarding accelera- Minimum Mean Square Error(MMSE)estimator. tions on the horizontal plane,to avoid the interferences such To perform space synchronization among multiple devices, as the up-and-down accelerations from the vertical direction MOSS uses a server to communicate with these devices and during human motions,we project the linear acceleration fi(t) process the corresponding measurements.We can set one of onto the horizontal plane in the reference local coordinate the mobile devices such as a smart phone as the server,as long system,as shown in Fig.8. as it can communicate with the other devices via wireless connections,and it has enough computing power for signal B.Principal Component Extraction processing.MOSS consists of the following four steps The acceleration measurements of different axes on multiple (1)Beacon Broadcasting:The server first broadcasts a devices are highly correlated,as shown in Eq.(1),because synchronization beacon to all devices.All devices start to of the consistent acceleration fe(t).This can be observed
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2151 Fig. 7. System architecture. where Ω(t) = ⎡ ⎣ 0 −ωz(t) ωy(t) ωz(t) 0 −ωx(t) −ωy(t) ωx(t) 0 ⎤ ⎦ (6) We further use Taylor expansion of the exponential term to approximate the value of exp( t+δt t Ω(t)dt). As the angular velocity signals obtained from the gyroscopes are “integrated” to track and update the rotation matrix, the errors in the gyroscope signals propagate into the calculated orientation leading to large cumulative tracking errors. The tracking errors in the rotation matrix At,t+δt are amplified in each axis through the exponential function in Eq.(5). The larger the angular velocities and linear accelerations in the forward moving direction are, the more the tracking errors ex(t), ey(t), and ez(t) in that direction. Thus, the rotation matrix A t,t+δt derived from gyroscope tracking is prone to large accumulation of errors after long periods. It is essential to leverage other measurements (such as the accelerometer measurements) for further calibration. V. SYSTEM OVERVIEW The system architecture is shown in Fig. 7. MOSS consists of two components: a Consistent Direction Estimator for achieving space synchronization, and a Gyroscope-based Orientation Tracker for maintaining space synchronization. Consistent Direction Estimator uses principal component analysis to extract the magnitude of consistent acceleration from multiple devices, and further estimate the direction of consistent acceleration. Combined with the gravity direction extracted from the acceleration measurements, it generates a rotation matrix for space synchronization. Gyroscope-based Orientation Tracker continuously estimates the orientation variation of the body frame from the gyroscope measurements. Leveraging the stability of the gravitational accelerations in the global coordinate, we calibrate the rotation matrix using the Minimum Mean Square Error (MMSE) estimator. To perform space synchronization among multiple devices, MOSS uses a server to communicate with these devices and process the corresponding measurements. We can set one of the mobile devices such as a smart phone as the server, as long as it can communicate with the other devices via wireless connections, and it has enough computing power for signal processing. MOSS consists of the following four steps. (1) Beacon Broadcasting: The server first broadcasts a synchronization beacon to all devices. All devices start to record the accelerometer and gyroscope measurements after receiving this beacon. Let this time be t0. (2) Local Measurement with Gyroscope Tracking: For each time t within a time window W, each device Di extracts the gravitational acceleration gi(t) and the linear acceleration fi(t) from the acceleration measurements in its local coordinate. Meanwhile, each device Di tracks the rotation speed ωi(t) in each axis from the gyroscope, and computes the rotation matrix Ai t0,t from time t0 to t. After time window W, device Di sends the following to the server: the rotation matrix Ai t0,t, the gravity gi(t), and the linear acceleration fi(t) for all t ∈ W. (3) PCA Analysis: After receiving the responses from all devices, the server performs PCA to extract the magnitude and estimates the direction of the consistent acceleration fc(t) in the local coordinate of each device. With the gravity direction, it constructs a rotation matrix Ci t0 for each device at time t0, and sends Ci t0 to each device. (4) Local Rotation Update: Each device Di locally updates the rotation matrix Ci t using the matrix Ai t0,t from gyroscope tracking, i.e., Ci t = Ci t0Ai t0,t. It then transforms the accelerometer and gyroscope measurements from the local coordinate to global coordinate. VI. CONSISTENT ACCELERATION EXTRACTION As the accelerometer measurements from multi-devices are highly correlated, we perform Principal Component Analysis (PCA) to extract the consistent acceleration during the process of moving forward. This consists of three steps: preprocessing, principal component extraction, and direction estimation. A. Preprocessing Suppose the acceleration measurements fi(t) for each device Di can be obtained as xi(t), yi(t), zi(t) from the x, y, and z-axes in its local coordinate Li(t) at time t. Then, we can use the rotation matrix (Ai t0,t)T to transform the acceleration measurements from the current local coordinate Li(t) at time t to the reference local coordinate Li(t0) at time t0, i.e., ⎡ ⎣ xi(t0) yi(t0) zi(t0) ⎤ ⎦ = (Ai t0,t) T ⎡ ⎣ xi(t) yi(t) zi(t) ⎤ ⎦ (7) Thus, we obtain the acceleration measurements in the reference local coordinate, i.e., the local coordinate at time t0, which is therefore consistent over time. Further, as we focus on extracting the forwarding accelerations on the horizontal plane, to avoid the interferences such as the up-and-down accelerations from the vertical direction during human motions, we project the linear acceleration fi(t) onto the horizontal plane in the reference local coordinate system, as shown in Fig. 8. B. Principal Component Extraction The acceleration measurements of different axes on multiple devices are highly correlated, as shown in Eq.(1), because of the consistent acceleration fc(t). This can be observed
2152 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 /Z axis linear acceleration fi(t) fiv(t):vertical projection of fi(t fiunt):horizontal n3a X axis Horizontal projection of fi(t) 250 graytational plane F npn1o acceleration gi(t) Y axis Fig.8.Project acceleration fi(t)onto horizontal plane Fig.10.PCA result vs.ground truth C.Direction Estimation 15 After principal component extraction.we obtain the magni- 复oWN tude of the consistent acceleration f(t),as shown in Fig.10. It is known that f(t)has a fixed direction,i.e..the forwarding direction,in regard to the reference local coordinate.During 50 00 50 200 25 Sampling Points(50Hz) the process of human body movements,the magnitude of fe(t). Fig.9.Correlations among 18 streams of acceleration measurements from i.e.,fe(t),keeps changing along or against the fixed direction. 6 devices in 3 axes. Hence,if we specify the magnitude fe(t)along the fixed direction as positive value,then the magnitude fe(t)against from Fig.9,which plots the 18 streams of acceleration the fixed direction can be specified as negative value.Next, measurements from 6 devices in 3 axes within a time inter- for each device D,,we need to further estimate the direction val of 5 seconds.Thus,to effectively extract the consistent of f(t)in its local coordinate. acceleration fe(t).we apply PCA to discover the correlation According to Theorem 1,the expected value of the incon- among the streams of acceleration measurements.With PCA, sistent acceleration from intra-body movement,i.e.,f(t). we can track the time-varying correlations among the streams is approximately equal to 0 in a sufficient large time interval. of acceleration measurements,and extract the principal com- It can be regarded as "white noises"in comparison to the ponents of these streams.This includes the following three consistent acceleration from the forwarding movement,since steps. the former is usually orders of magnitude smaller than the (1)Preprocessing:Suppose the number of devices is m, latter.Hence,in order to extract the components of consistent we organize the data set as a matrix X consisting of p acceleration in various axes,while eliminating the interference streams of acceleration measurements in 3 axes of local from the inconsistent acceleration,we can select a sufficiently coordinate from m devices,where p =3 x m.Each column large time interval,add up the acceleration measurements for of the matrix X corresponds to one stream of acceleration each axis according to Eq.(1),and obtain the estimate of each measurements within a sampling window of size n.We component in the corresponding axis.However,we need to choose the default time interval of the sampling window to be tackle the following issue:As shown in Fig.10,while the 10 seconds as the sampling process should not be too long for human subject is moving forward,the consistent acceleration human motions and meanwhile the number of samples is large fe(t)keeps changing around the zero point even during one enough to ensure accurate correlation estimation.Therefore. stride.Nevertheless,on the macro level,the moving speed of the matrix X is of dimensions n x p.Our solution first human subject usually keeps constant or does not change too calculates the empirical mean i along each column of the much.This implies that the expected value of fe(t)should data matrix X.and then subtracts the empirical mean vector i also be close to 0(it is in the order of 10-310-2 m/s2 from each column(i=l···p). according to the empirical study),when a human subject is (2)Covariance Estimation:We calculate the covariance moving in almost constant speed.In this regard,even if the matrix as XTx X.The covariance matrix has the dimension consistent acceleration fe(t)has a much larger magnitude of p x p,where p is the number of streams for acceleration than the inconsistent acceleration f(t),when we add up the measurements.Then,we obtain the eigenvectors and eigen- consistent accelerations,the positive and negative components values of the covariance matrix,and construct the principal cancel each other to approach a rather small value,which components using the equation j=Xx gi,where g and zj prevents us to distinguish the consistent accelerations from are the jth eigenvector and the jth principal components, inconsistent accelerations. respectively. Fortunately,since we can obtain the magnitude of fe(t), (3)Consistent Forwarding Acceleration Extraction:We i.e.,fe(t),via principal component extraction,by using the select the first principal component zi to estimate the mag- techniques such as linear interpolation,we can identify the nitude of the consistent acceleration.We further plot the time intervals where the value of fe(t)is positive,we call PCA results in Fig.10 to compare with the ground truth of these time intervals positive time intervals.Similarly,we can consistent acceleration.According to the experiment results, also identify the negative time intervals where the value of the correlation between the PCA result and the ground truth fe(t)is negative.Fig.11 shows an example of identifying is up to 0.961,which implies that we can accurately extract the positive time interval according to the PCA results.Then, the consistent acceleration using PCA. for each axis,without loss of generality,we can add up
2152 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 Fig. 8. Project acceleration fi(t) onto horizontal plane. Fig. 9. Correlations among 18 streams of acceleration measurements from 6 devices in 3 axes. from Fig. 9, which plots the 18 streams of acceleration measurements from 6 devices in 3 axes within a time interval of 5 seconds. Thus, to effectively extract the consistent acceleration fc(t), we apply PCA to discover the correlation among the streams of acceleration measurements. With PCA, we can track the time-varying correlations among the streams of acceleration measurements, and extract the principal components of these streams. This includes the following three steps. (1) Preprocessing: Suppose the number of devices is m, we organize the data set as a matrix X consisting of p streams of acceleration measurements in 3 axes of local coordinate from m devices, where p = 3 × m. Each column of the matrix X corresponds to one stream of acceleration measurements within a sampling window of size n. We choose the default time interval of the sampling window to be 10 seconds as the sampling process should not be too long for human motions and meanwhile the number of samples is large enough to ensure accurate correlation estimation. Therefore, the matrix X is of dimensions n × p. Our solution first calculates the empirical mean μi along each column of the data matrix X, and then subtracts the empirical mean vector μi from each column (i = 1 ··· p). (2) Covariance Estimation: We calculate the covariance matrix as XT × X. The covariance matrix has the dimension of p × p, where p is the number of streams for acceleration measurements. Then, we obtain the eigenvectors and eigenvalues of the covariance matrix, and construct the principal components using the equation xj = X ×qj , where qj and xj are the jth eigenvector and the jth principal components, respectively. (3) Consistent Forwarding Acceleration Extraction: We select the first principal component x1 to estimate the magnitude of the consistent acceleration. We further plot the PCA results in Fig. 10 to compare with the ground truth of consistent acceleration. According to the experiment results, the correlation between the PCA result and the ground truth is up to 0.961, which implies that we can accurately extract the consistent acceleration using PCA. Fig. 10. PCA result vs. ground truth. C. Direction Estimation After principal component extraction, we obtain the magnitude of the consistent acceleration fc(t), as shown in Fig. 10. It is known that fc(t) has a fixed direction, i.e., the forwarding direction, in regard to the reference local coordinate. During the process of human body movements, the magnitude of fc(t), i.e., fc(t), keeps changing along or against the fixed direction. Hence, if we specify the magnitude fc(t) along the fixed direction as positive value, then the magnitude fc(t) against the fixed direction can be specified as negative value. Next, for each device Di, we need to further estimate the direction of fc(t) in its local coordinate. According to Theorem 1, the expected value of the inconsistent acceleration from intra-body movement, i.e., f i (t), is approximately equal to 0 in a sufficient large time interval. It can be regarded as “white noises” in comparison to the consistent acceleration from the forwarding movement, since the former is usually orders of magnitude smaller than the latter. Hence, in order to extract the components of consistent acceleration in various axes, while eliminating the interference from the inconsistent acceleration, we can select a sufficiently large time interval, add up the acceleration measurements for each axis according to Eq.(1), and obtain the estimate of each component in the corresponding axis. However, we need to tackle the following issue: As shown in Fig. 10, while the human subject is moving forward, the consistent acceleration fc(t) keeps changing around the zero point even during one stride. Nevertheless, on the macro level, the moving speed of human subject usually keeps constant or does not change too much. This implies that the expected value of fc(t) should also be close to 0 (it is in the order of 10−3 ∼ 10−2 m/s2 according to the empirical study), when a human subject is moving in almost constant speed. In this regard, even if the consistent acceleration fc(t) has a much larger magnitude than the inconsistent acceleration f i(t), when we add up the consistent accelerations, the positive and negative components cancel each other to approach a rather small value, which prevents us to distinguish the consistent accelerations from inconsistent accelerations. Fortunately, since we can obtain the magnitude of fc(t), i.e., fc(t), via principal component extraction, by using the techniques such as linear interpolation, we can identify the time intervals where the value of fc(t) is positive, we call these time intervals positive time intervals. Similarly, we can also identify the negative time intervals where the value of fc(t) is negative. Fig. 11 shows an example of identifying the positive time interval according to the PCA results. Then, for each axis, without loss of generality, we can add up
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2153 Positive Time Interval 80 20 Axs:g' Ag8:日 Fig.11.Identify the positive time interval according to the PCA results. Fig.12.Estimate the rotation matrix At.t+6t. those acceleration measurements only from the positive time intervals to extract the accumulated positive magnitudes of According to Theorem 2,our solution of estimating the the consistent acceleration fe(t).According to the formulation direction of fe(t)is as follows:First,we select a sufficiently in Eq.(1),for the accumulated acceleration measurements, large time interval to obtain the corresponding PCA results, the accumulated positive magnitudes of f(t)dominates the and identify the positive time intervals from the PCA results result than the accumulated magnitudes of the inconsistent where the accelerations are greater than 0.Then,for each axis acceleration f(t),which is very close to 0.In this way, of the reference local coordinate,we compute the expected we are able to further estimate the direction of fc(t)in value of the acceleration measurements within the above posi- its local coordinate,according to its positive magnitudes in tive time intervals.After that,we can compute the angles oi,Bi different axes.Therefore,we can estimate the direction of()andaccording to Eq.(),and estimate the directionoff(t). according to Theorem 2. As the value of E[fe(t)]is the same in all three formulations Theorem 2:Let i,Bi and Yi be the constant angles in Eq.(8),it is actually unnecessary to compute the value between the consistent acceleration fe(t)and the z,y and z of E[fe(t)].As a matter of fact,according to the previous axes in the reference local coordinate,respectively.Let fe(t) analysis,we have vE[fi(t)]2 +E[fi.(t)]2 +E[fi.2(t)2 be the magnitude of fe(t),and fi(t),fi.(t)and fi.=(t)be E[fe(t)]. the acceleration measurements in the z,y,and z-axes from the accelerometer,respectively.The direction of f(t)can be VII.GYROSCOPE BASED ORIENTATION TRACKING represented by a unit vector (cos ai,cos Bi,cos i),where the A.Rotation Based Modeling value of cos ai,cos Bi,and cosYi can be estimated as follows: Our solution is based on the observation that the extracted E[fi.(t)] cos ai= direction of gravity is stable in the global coordinate.This ELf(t切 observation can be further leveraged to calibrate the rotation cos Bi= E[fi,y(t】 (8) matrix A.t+t derived from gyroscope based tracking.For E[fe(t)] device Di,suppose that the gravity is denoted as a vector E[fi.z(t)] g=(gz(t),gu(t),g=(t))in the local coordinate system Li(t) cos= E[fe(t)] at time t.Then,after a period of tracking,the gravity could be denoted as a different vector g'=(gz(t +6t),gu(t+ Proof:Without loss of generality,we prove the equation EIfi.z(t)] 6t),g2(t +6t))in the local coordinate system Li(t +6t)at cosai According to Eg.(1).for the z-axis, DLfe(t打 time t+6t.The difference between g and g'is caused by fi.(t)=fe(t)cosi+f(t).After we obtain a sequence of the rotation of the body frame from time t to time t+ot. fi.(t)from the positive time intervals in a sufficiently large we denote the corresponding rotation matrix as A.t+.Thus time window Wp,the sum of fi.(t)in these positive time we have g'=At.t+6g.According to the above relationship, intervals,ie..tew(t).can be depicted as follows: the rotation matrix At.t+6t can be computed as follows.As ∑fi()=∑[f():cos剑+∑fi(t). shown in Fig.12,note that due to the rotation of the body frame,there may exist a non-zero angle between the vectors tEWp tEWp g'and g.it can be calculated as:-arccos().where According to Theorem 1.tewf(t)0.Besides, the.operation refers to the inner product of the two vectors. since fe(t)is extracted from the positive time intervals,then During the process of rotation,to align the vector g to the ∑tew,f.()·cos]≥∑tew,fi.().Thus, vector g',it is essential to rotate axis u by an angle 0.The rotation axis u is orthogonal to the plane where the vectors ∑f.()≈∑[f.()cos创. g and g'lies on.Then,the rotation axis u can be obtained tEW by computing the cross product of g and g':u EXg' g×gT Moreover,since cosa;is constant,then, We call the above rotation as the vertical rotation.Therefore, ∑tew,fiz( according to the Rodrigues'rotation formula [28].we compute Cos≈ E[fi.=(t)] the corresponding rotation matrix R as: ∑tew,f.(O E[fe(t)] R=I+(sin0)U+(1-cos0)U2 (9) Similarly,we can prove that cos Elfi(t)] and Efe(t) Ef.=(t)】 where I is a 3 x 3 unit matrix,and U denote the cross-product c0s7= E fe(t) matrix for vector u
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2153 Fig. 11. Identify the positive time interval according to the PCA results. those acceleration measurements only from the positive time intervals to extract the accumulated positive magnitudes of the consistent acceleration fc(t). According to the formulation in Eq.(1), for the accumulated acceleration measurements, the accumulated positive magnitudes of fc(t) dominates the result than the accumulated magnitudes of the inconsistent acceleration f i(t), which is very close to 0. In this way, we are able to further estimate the direction of fc(t) in its local coordinate, according to its positive magnitudes in different axes. Therefore, we can estimate the direction of fc(t) according to Theorem 2. Theorem 2: Let αi, βi and γi be the constant angles between the consistent acceleration fc(t) and the x, y and z axes in the reference local coordinate, respectively. Let fc(t) be the magnitude of fc(t), and fi,x(t), fi,y(t) and fi,z(t) be the acceleration measurements in the x, y, and z-axes from the accelerometer, respectively. The direction of fc(t) can be represented by a unit vector cos αi, cos βi, cos γi, where the value of cos αi, cos βi, and cos γi can be estimated as follows: ⎧ ⎪⎪⎪⎪⎪⎪⎨ ⎪⎪⎪⎪⎪⎪⎩ cos α i = E[fi,x(t)] E[fc(t)] cos β i = E[fi,y(t)] E[fc(t)] cos γ i = E[fi,z(t)] E[fc(t)] . (8) Proof: Without loss of generality, we prove the equation cos α i = E[fi,x(t)] E[fc(t)] . According to Eq.(1), for the x-axis, fi,x(t) = fc(t) cos α i +f i,x(t). After we obtain a sequence of fi,x(t) from the positive time intervals in a sufficiently large time window Wp, the sum of fi,x(t) in these positive time intervals, i.e., t∈Wp fi,x(t), can be depicted as follows: t∈Wp fi,x(t) = t∈Wp [fc(t) · cos α i] + t∈Wp f i,x(t). According to Theorem 1, t∈Wp f i,x(t) ≈ 0. Besides, since fc(t) is extracted from the positive time intervals, then t∈Wp [fc(t) · cos α i] t∈Wp f i,x(t). Thus, t∈Wp fi,x(t) ≈ t∈Wp [fc(t) · cos α i]. Moreover, since cos α i is constant, then, cos α i ≈ t∈Wp fi,x(t) t∈Wp fc(t) ≈ E[fi,x(t)] E[fc(t)] . Similarly, we can prove that cos β i = E[fi,y(t)] E[fc(t)] and cos γ i = E[fi,z(t)] E[fc(t)] . Fig. 12. Estimate the rotation matrix At,t+δt. According to Theorem 2, our solution of estimating the direction of fc(t) is as follows: First, we select a sufficiently large time interval to obtain the corresponding PCA results, and identify the positive time intervals from the PCA results where the accelerations are greater than 0. Then, for each axis of the reference local coordinate, we compute the expected value of the acceleration measurements within the above positive time intervals. After that, we can compute the angles αi, βi and γi according to Eq. (8), and estimate the direction of fc(t). As the value of E[fc(t)] is the same in all three formulations in Eq. (8), it is actually unnecessary to compute the value of E[fc(t)]. As a matter of fact, according to the previous analysis, we have E[fi,x(t)]2 + E[fi,y(t)]2 + E[fi,z(t)]2 ≈ E[fc(t)]. VII. GYROSCOPE BASED ORIENTATION TRACKING A. Rotation Based Modeling Our solution is based on the observation that the extracted direction of gravity is stable in the global coordinate. This observation can be further leveraged to calibrate the rotation matrix A t,t+δt derived from gyroscope based tracking. For device Di, suppose that the gravity is denoted as a vector g = gx(t), gy(t), gz(t) in the local coordinate system Li(t) at time t. Then, after a period of tracking, the gravity could be denoted as a different vector g’ = gx(t + δt), gy(t + δt), gz(t + δt) in the local coordinate system Li(t + δt) at time t + δt. The difference between g and g’ is caused by the rotation of the body frame from time t to time t + δt, we denote the corresponding rotation matrix as At,t+δt. Thus we have g’ = At,t+δtg. According to the above relationship, the rotation matrix At,t+δt can be computed as follows. As shown in Fig. 12, note that due to the rotation of the body frame, there may exist a non-zero angle θ between the vectors g’ and g, it can be calculated as: θ = arccos( g·g’ g|g’ ), where the · operation refers to the inner product of the two vectors. During the process of rotation, to align the vector g to the vector g’, it is essential to rotate axis u by an angle θ. The rotation axis u is orthogonal to the plane where the vectors g and g’ lies on. Then, the rotation axis u can be obtained by computing the cross product of g and g’: u = g×g’ g×g’ . We call the above rotation as the vertical rotation. Therefore, according to the Rodrigues’ rotation formula [28], we compute the corresponding rotation matrix R as: R = I + (sin θ)U + (1 − cos θ)U2 (9) where I is a 3×3 unit matrix, and U denote the cross-product matrix for vector u.
2154 IEEE/ACM TRANSACTIONS ON NETWORKING,VOL.26.NO.5.OCTOBER 2018 By now we have obtained the rotation matrix by rotating the experience a consistent acceleration.We address this issue body frame to align the previous gravity vector g to the current based on the observation that if multiple devices are subject gravity vector g'in the vertical direction.However,there still to a consistent accelerations,then all the accelerometers of exists another degree of freedom in the horizontal direction the devices should experience similar accelerations at a coarse because the body frame can rotate about the current gravity level [29],[30,as the accelerations due to human motions are vector g'by any angle 0'on the horizontal plane.Thus, likely to dominate the accelerations caused by other intra-body we need to further compute the rotation matrix R'with the motions.Thus,for each pair of devices,we compute the cor- rotation angle on the horizontal plane to accurately estimate relation of the measured accelerations at 1-10 Hz frequencies, the rotation matrix At.t+6t.As the body frame is rotated which are the frequency range of human motions.We use a around the current gravity vector g',then the rotation axis Support Vector Machine based classifier to determine whether can be defined by a unit vector u'by normalizing the gravity all devices are subject to a consistent acceleration according vector g',i.e.,u'=According to the Rodrigues'rotation to their pair-wise correlations. formula,suppose the rotation angle in the horizontal plane Direction Change of Consistent Acceleration:During the is 0',we compute the corresponding rotation matrix R'()as process of human motions,the consistent acceleration could be follows: also changing by the human subject over time.The change of R'(@')=I+(sin')U'+(1-cos0')U2 the direction could be performed intentionally when the human (10) subject is turning around or unintentionally when the human where I is a 3x3 unit matrix,and U'denote the cross-product subject is slightly off the straight path.In such situations, matrix for the vector u'.We call the above rotation as the it is essential to synchronize the global frames derived from horizontal rotation.Therefore,the rotation matrix At.t+6t can the consistent acceleration with different directions:otherwise. be estimated as follows: the global coordinates cannot be consistent with each other. We address this issue as follows:Suppose that all devices are At.:+61(0')=RR'(0') (11) subject to two different forwarding accelerations with different Fig.12 illustrates the two rotations for estimating the rotation directions at time tk-1 and tk,respectively.After PCA. matrix At.t+6t. for each device.we obtain the two forwarding accelerations based on the reference local coordinate,i.e.,fe()and B.Calibration With MMSE Estimator fe(tk),respectively.We calculate the angle deviation between fe(tk-1)and fe(tk)on the horizontal plane as ni for each According to the observations in previous work [6].[7] device.After that,we compute the average value from all and the analysis in Section 4,we know that after a long m devices to mitigate the variances in estimating the angle period of tracking,there could be a fairly large error for deviation.Then,we use the angle deviation n as the direction the estimator At.t+6t of gyroscope tracking.Fortunately. change of the consistent acceleration to further calibrate the we have already calculated a rotation matrix At.t+6t() rotation matrix for space synchronization. from the rotation-based model.As the gravity direction can be accurately extracted through low pass filters such as the B.Limitations Butterworth filter,we actually obtain an accurate formulation Limitation to the Potential Application Scenarios:MOSS of the rotation matrix At.t+6t with an unknown variable 0'. uses an estimator to derive a consistent direction on the Thus,to estimate the rotation angle according to the horizontal plane,such as the moving direction.Combined estimator At.t+6t from the gyroscope tracking,we leverage the with the gravity direction,MOSS build synchronized coor- Minimal Mean Squared Error (MMSE)estimator.We define dinates among different mobile devices.Hence,a consistent the estimation error matrix as E=At.t+6t-At.t+6t,where acceleration on the horizontal plane for multiple devices is the mean squared error(MSE)e is: essential for space synchronization.This requirement may not 3 be satisfied for all applications.In some applications,e.g., （E()2=∑∑ (A-A(0)2 when a human subject is jumping up and down with multiple i=1j= devices,the consistent acceleration only exist on the vertical We then compute the optimal value of that achieves the plane,which does not help to build synchronized coordinates minimal mean squared error for the estimator At.t+6t: among these devices.Besides,even if there exists a consistent acceleration for multiple devices on the horizontal plane,when 0*=argminge(') (12) the consistent acceleration changes from time to time in a Thus,after computing the optimal value 0',we use the frequent manner,the time window to collect the samplings of rotation matrix At.t+t()as the calibrated estimator for the the acceleration might be too small,such that the consistent acceleration cannot be effectively extracted from the limited rotation matrix At.t+6t. time window.Thus,the application scope of MOSS does have VIII.DISCUSSIONS AND LIMITATIONS some limitations in the property of consistent acceleration. Vulnerability to Sudden Activities:MOSS achieves good A.Discussions performance when the human subject is performing regular Timing for Space Synchronization:During the process of free activities,such as walking,jumping,and running,in which human motions.we need to find the time that all devices the arm's or hand's movements in these activities are relatively
2154 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 26, NO. 5, OCTOBER 2018 By now we have obtained the rotation matrix by rotating the body frame to align the previous gravity vector g to the current gravity vector g’ in the vertical direction. However, there still exists another degree of freedom in the horizontal direction because the body frame can rotate about the current gravity vector g’ by any angle θ on the horizontal plane. Thus, we need to further compute the rotation matrix R’ with the rotation angle θ on the horizontal plane to accurately estimate the rotation matrix At,t+δt. As the body frame is rotated around the current gravity vector g’, then the rotation axis can be defined by a unit vector u’ by normalizing the gravity vector g’, i.e., u’ = g’ g’ . According to the Rodrigues’ rotation formula, suppose the rotation angle in the horizontal plane is θ , we compute the corresponding rotation matrix R’(θ ) as follows: R (θ ) = I + (sin θ )U + (1 − cos θ )U2 (10) where I is a 3×3 unit matrix, and U denote the cross-product matrix for the vector u’. We call the above rotation as the horizontal rotation. Therefore, the rotation matrix A t,t+δt can be estimated as follows: At,t+δt(θ ) = RR (θ ) (11) Fig. 12 illustrates the two rotations for estimating the rotation matrix A t,t+δt. B. Calibration With MMSE Estimator According to the observations in previous work [6], [7] and the analysis in Section 4, we know that after a long period of tracking, there could be a fairly large error for the estimator A t,t+δt of gyroscope tracking. Fortunately, we have already calculated a rotation matrix At,t+δt(θ ) from the rotation-based model. As the gravity direction can be accurately extracted through low pass filters such as the Butterworth filter, we actually obtain an accurate formulation of the rotation matrix At,t+δt with an unknown variable θ . Thus, to estimate the rotation angle θ according to the estimator A t,t+δt from the gyroscope tracking, we leverage the Minimal Mean Squared Error (MMSE) estimator. We define the estimation error matrix as E = A t,t+δt − At,t+δt, where the mean squared error (MSE) e is: e(θ ) = 3 i=1 3 j=1 (Ei,j (θ ))2 = 3 i=1 3 j=1 (A i,j − Ai,j (θ ))2 We then compute the optimal value of θ that achieves the minimal mean squared error for the estimator A t,t+δt: θ∗ = argminθe(θ ) (12) Thus, after computing the optimal value θ∗, we use the rotation matrix At,t+δt(θ∗) as the calibrated estimator for the rotation matrix At,t+δt. VIII. DISCUSSIONS AND LIMITATIONS A. Discussions Timing for Space Synchronization: During the process of human motions, we need to find the time that all devices experience a consistent acceleration. We address this issue based on the observation that if multiple devices are subject to a consistent accelerations, then all the accelerometers of the devices should experience similar accelerations at a coarse level [29], [30], as the accelerations due to human motions are likely to dominate the accelerations caused by other intra-body motions. Thus, for each pair of devices, we compute the correlation of the measured accelerations at 1-10 Hz frequencies, which are the frequency range of human motions. We use a Support Vector Machine based classifier to determine whether all devices are subject to a consistent acceleration according to their pair-wise correlations. Direction Change of Consistent Acceleration: During the process of human motions, the consistent acceleration could be also changing by the human subject over time. The change of the direction could be performed intentionally when the human subject is turning around or unintentionally when the human subject is slightly off the straight path. In such situations, it is essential to synchronize the global frames derived from the consistent acceleration with different directions; otherwise, the global coordinates cannot be consistent with each other. We address this issue as follows: Suppose that all devices are subject to two different forwarding accelerations with different directions at time tk−1 and tk, respectively. After PCA, for each device, we obtain the two forwarding accelerations based on the reference local coordinate, i.e., fc(tk−1) and fc(tk), respectively. We calculate the angle deviation between fc(tk−1) and fc(tk) on the horizontal plane as ηi for each device. After that, we compute the average value η¯ from all m devices to mitigate the variances in estimating the angle deviation. Then, we use the angle deviation η¯ as the direction change of the consistent acceleration to further calibrate the rotation matrix for space synchronization. B. Limitations Limitation to the Potential Application Scenarios: MOSS uses an estimator to derive a consistent direction on the horizontal plane, such as the moving direction. Combined with the gravity direction, MOSS build synchronized coordinates among different mobile devices. Hence, a consistent acceleration on the horizontal plane for multiple devices is essential for space synchronization. This requirement may not be satisfied for all applications. In some applications, e.g., when a human subject is jumping up and down with multiple devices, the consistent acceleration only exist on the vertical plane, which does not help to build synchronized coordinates among these devices. Besides, even if there exists a consistent acceleration for multiple devices on the horizontal plane, when the consistent acceleration changes from time to time in a frequent manner, the time window to collect the samplings of the acceleration might be too small, such that the consistent acceleration cannot be effectively extracted from the limited time window. Thus, the application scope of MOSS does have some limitations in the property of consistent acceleration. Vulnerability to Sudden Activities: MOSS achieves good performance when the human subject is performing regular free activities, such as walking, jumping, and running, in which the arm’s or hand’s movements in these activities are relatively
XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2155 simple and regular.However,in more general activities,e.g., the activities for VR applications,in which the motions can lead to sudden,rapid and irregular acceleration and orientation changes,MOSS may not achieve ideal performance for the following reasons:1)In sudden activities,the inconsistent accelerations can be comparable to the consistent acceleration in the magnitude of the movement,such that the consistent 400 acceleration cannot be effectively extracted.Besides,as afore- Sampling Point (50Hz) mentioned,the time window for the consistent acceleration can Fig.13. Inertial readings with/without Kalman filter. be too small,this further increases the difficulty to extract the within a time window,respectively.Take the acceleration in consistent acceleration.2)Sudden activity may lead to large r-axis as an example,we use 1to denote the estimated errors in the inertial measurements including the accelerometer acceleration at time t-1.Then,we use Kalman filter to measurements and gyroscope measurements.The errors in estimate the true acceleration at time tk,while combining the accelerometer measurements further lead to difficulty in the last estimated acceleration k1 at time t-1 and the achieving space synchronization,whereas the errors in the measurement z from the accelerometer at time t.Kalman gyroscope measurements further lead to difficulty in maintain- filter is a recursive estimator,which contains a prediction phase ing space synchronization.Therefore,MOSS is more or less and an update phase in each recursive process. vulnerable to the sudden activities due to the above reasons. In the prediction phase,we use k-1lk1 to predict the next acceleration as klk-1,i.e.,Tklk-1 Tk-11k-1.The IX.PERFORMANCE EVALUATION estimate covariance Pk-1=P-1k-1+Q,where P-1k-1 A.Implementation and Setup means the estimate covariance in the last update phase,while O means the covariance of the process noise in the prediction Hardware:We implemented MOSS on 1 smart glasses phase.Here,the process noise is assumed as Gaussian white (Google Glass 2)and 5 smart phones(SAMSUNG Galaxy S5). They were placed on a human subject as shown in Fig.2. noise N(uQ,Q)(Q =1.0004 in our implementation). In the update phase,we combine the predicted acceleration These devices were connected to an Android smartphone (SAMSUNG Galaxy S5)via the Bluetooth 4.0 interface, k-1 and the measurement zk to update the estimate of the i.e.,S2 in Fig.2.The Android smartphone S2 continuously acceleration klk at time tk,i.e.,klk klk-1+Kk(zk- collected the accelerometer and gyroscope measurements from k-1).Here,K&means the Kalman gain at time tk,and it these devices,and processed these measurements. can be calculated as follows:Kk=Pk-1(Pk-1+R)-1, where R means the covariance of the measurement noise Setup:We let 5 volunteers move along three different in the update phase,and the measurement noise is assumed straight paths in outdoor environment.They moved along the paths in three different modes:1)walk:the human subject as Gaussian white noise N(uR,R)(R 0.95976 in our implementation).In the update phase,the estimate covariance walked in normal speed (0.5~1 m/s)with small-amplitudes; 2)run:the human subject ran in fairly fast speed(2~3m/s) Pk=(1-Kk)P-1.Therefore,the optimal estimation of with large-amplitudes;3)jump:the human subject jumped acceleration in r-axis at time tk is klk.Similarly,we can obtain the optimal estimation of the acceleration in y-axis. forward with large-amplitudes.For each path,we collected 20 traces for each moving mode.In each trace,we col- and z-axis at time t as ylk and k respectively.After that, lected the inertial measurements from the accelerometers. we average the values ofwithin a time window, gyroscopes,and magnetometers of all devices for a maximum respectively,to infer the forwarding direction.The solution is effective when the body frames of the devices do not change time interval of 30 seconds.We collected a total of 120 traces too much during the moving process. of inertial measurements.To validate the performance against the ground truth,we extracted the magnetic direction and 2)RefMeanAcc:it estimates the forwarding direction by averaging the acceleration measurements within a time win- gravity direction from the inertial measurement,and used them dow respectively in three axes of the reference local coordi- to build an earth coordinate system as the reference coordinate nate.The main difference between MeanAcc and RefMeanAcc system E. is that RefMeanAcc transforms all the acceleration from the Metrics:We used two metrics to evaluate MOSS:(1)Angle body frame to the reference local coordinate.After that, Accuracy:the angle deviation between the estimated forward- ing direction and the ground truth,(2)Coordinate Accuracy: RefMeanAcc calculates the acceleration in each axis based on Kalman filter [31]in the same way described in MeanAcc. the correlations between the measurements from the synchro- The solution is effective even when the body frames of the nized coordinate and ground truth. devices keep changing,using the gyroscope tracking Experimental results show that Kalman filter can effectively B.Forwarding Direction Estimation remove the outliers or mitigate the jitters of the raw iner- We compared our solution with two heuristic solutions using tial readings.Fig.13 shows the corresponding inertial readings Kalman filter as baseline solutions.i.e.. with or without the Kalman filter.Note that in comparison 1)MeanAcc:it uses Kalman filter [31]to filter the inertial to the raw inertial measurement,the inertial readings after readings first,and then estimates the forwarding direction by the Kalman filter are more smooth and have less jitters.It averaging the acceleration in three axes of the body frame implies that,for the situations where the human motions lead
XIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2155 simple and regular. However, in more general activities, e.g., the activities for VR applications, in which the motions can lead to sudden, rapid and irregular acceleration and orientation changes, MOSS may not achieve ideal performance for the following reasons: 1) In sudden activities, the inconsistent accelerations can be comparable to the consistent acceleration in the magnitude of the movement, such that the consistent acceleration cannot be effectively extracted. Besides, as aforementioned, the time window for the consistent acceleration can be too small, this further increases the difficulty to extract the consistent acceleration. 2) Sudden activity may lead to large errors in the inertial measurements including the accelerometer measurements and gyroscope measurements. The errors in the accelerometer measurements further lead to difficulty in achieving space synchronization, whereas the errors in the gyroscope measurements further lead to difficulty in maintaining space synchronization. Therefore, MOSS is more or less vulnerable to the sudden activities due to the above reasons. IX. PERFORMANCE EVALUATION A. Implementation and Setup Hardware: We implemented MOSS on 1 smart glasses (Google Glass 2) and 5 smart phones (SAMSUNG Galaxy S5). They were placed on a human subject as shown in Fig. 2. These devices were connected to an Android smartphone (SAMSUNG Galaxy S5) via the Bluetooth 4.0 interface, i.e., S2 in Fig. 2. The Android smartphone S2 continuously collected the accelerometer and gyroscope measurements from these devices, and processed these measurements. Setup: We let 5 volunteers move along three different straight paths in outdoor environment. They moved along the paths in three different modes: 1) walk: the human subject walked in normal speed (0.5∼1 m/s) with small-amplitudes; 2) run: the human subject ran in fairly fast speed (2∼3m/s) with large-amplitudes; 3) jump: the human subject jumped forward with large-amplitudes. For each path, we collected 20 traces for each moving mode. In each trace, we collected the inertial measurements from the accelerometers, gyroscopes, and magnetometers of all devices for a maximum time interval of 30 seconds. We collected a total of 120 traces of inertial measurements. To validate the performance against the ground truth, we extracted the magnetic direction and gravity direction from the inertial measurement, and used them to build an earth coordinate system as the reference coordinate system E. Metrics: We used two metrics to evaluate MOSS: (1) Angle Accuracy: the angle deviation between the estimated forwarding direction and the ground truth, (2) Coordinate Accuracy: the correlations between the measurements from the synchronized coordinate and ground truth. B. Forwarding Direction Estimation We compared our solution with two heuristic solutions using Kalman filter as baseline solutions, i.e., 1) MeanAcc: it uses Kalman filter [31] to filter the inertial readings first, and then estimates the forwarding direction by averaging the acceleration in three axes of the body frame Fig. 13. Inertial readings with/without Kalman filter. within a time window, respectively. Take the acceleration in x-axis as an example, we use xk−1|k−1 to denote the estimated acceleration at time tk−1. Then, we use Kalman filter to estimate the true acceleration xk|k at time tk, while combining the last estimated acceleration xk−1|k−1 at time tk−1 and the measurement zk from the accelerometer at time tk. Kalman filter is a recursive estimator, which contains a prediction phase and an update phase in each recursive process. In the prediction phase, we use xk−1|k−1 to predict the next acceleration as xk|k−1, i.e., xk|k−1 = xk−1|k−1. The estimate covariance Pk|k−1 = Pk−1|k−1 +Q, where Pk−1|k−1 means the estimate covariance in the last update phase, while Q means the covariance of the process noise in the prediction phase. Here, the process noise is assumed as Gaussian white noise N(μQ, Q)(Q = 1.0004 in our implementation). In the update phase, we combine the predicted acceleration xk|k−1 and the measurement zk to update the estimate of the acceleration xk|k at time tk, i.e., xk|k = xk|k−1 + Kk(zk − xk|k−1). Here, Kk means the Kalman gain at time tk, and it can be calculated as follows: Kk = Pk|k−1(Pk|k−1 + R)−1, where R means the covariance of the measurement noise in the update phase, and the measurement noise is assumed as Gaussian white noise N(μR, R) (R = 0.95976 in our implementation). In the update phase, the estimate covariance Pk|k = (1 − Kk)Pk|k−1. Therefore, the optimal estimation of acceleration in x-axis at time tk is xk|k. Similarly, we can obtain the optimal estimation of the acceleration in y-axis, and z-axis at time tk as yk|k and zk|k, respectively. After that, we average the values of xk|k, yk|k, zk|k within a time window, respectively, to infer the forwarding direction. The solution is effective when the body frames of the devices do not change too much during the moving process. 2) RefMeanAcc: it estimates the forwarding direction by averaging the acceleration measurements within a time window respectively in three axes of the reference local coordinate. The main difference between MeanAcc and RefMeanAcc is that RefMeanAcc transforms all the acceleration from the body frame to the reference local coordinate. After that, RefMeanAcc calculates the acceleration in each axis based on Kalman filter [31] in the same way described in MeanAcc. The solution is effective even when the body frames of the devices keep changing, using the gyroscope tracking. Experimental results show that Kalman filter can effectively remove the outliers or mitigate the jitters of the raw inertial readings. Fig. 13 shows the corresponding inertial readings with or without the Kalman filter. Note that in comparison to the raw inertial measurement, the inertial readings after the Kalman filter are more smooth and have less jitters. It implies that, for the situations where the human motions lead