XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2147 caused by the massive steel embedded in building concrete to these three axes.we can build the synchronized coordinates structures and other metallic objects.Our empirical study with each device as the origins. further shows that the magnetometer readings are not reli- Furthermore,we use gyroscope readings from multiple able in typical indoor environment.Specifically,for differ- mobile devices on a human subject to maintain space syn- ent positions in the indoor environment,the angle deviation chronization when the human subject stops moving forward, between the magnetometer measurement and the ground-truth which means that we can no longer obtain the consistent north can be as large as 110,the standard deviation of the acceleration caused by forward body motion.After the human magnetometer readings can be as large as 26.7;for a fixed subject stops moving forward,his body parts may still slightly position in the indoor environment,with the interferences from move or rotate.The gyroscope readings of a mobile device different sources such as smart phone,earphone,metal plate, allow us to continuously track the small rotations along and magnet,the angle deviation between the magnetometer the three axes of the device's local coordinate.We derive measurement and the ground-truth north can be as large as a real-time rotation matrix corresponding to the orientation 133.6,the standard deviation of the angle deviation can be variation by integrating the rotation rates in different axes over as large as 86.3 time.Leveraging the stability of the gravity direction in the synchronized coordinates,we further calibrate the estimated B.Proposed Approach rotation matrix from gyroscope tracking.Thus,we are able to In this paper,we propose a scheme called MObile Space maintain space synchronization even after the human subject Synchronization (MOSS)for devices with two sensors:an stops moving. accelerometer and a gyroscope,which are available on most mobile devices.Accelerometer readings from multiple mobile C.Technical Challenges and Solutions devices on a human subject are used to achieve space synchro- The first challenge is to extract the consistent acceleration nization when the human subject is moving forward,such as of body movement when the human subject moves forward. walking and running.Our insight on using accelerometer sen- This is challenging because both consistent and inconsistent sors to achieve space synchronization is that when the human accelerations are mixed together.As the mobile devices are subject moves forward,all attached mobile devices experience attached to different body locations of the human subject,they the same acceleration along the moving direction of the torso, perceive rather different accelerations in both the direction and which we call consistent acceleration.Although these mobile magnitude during human motion.To address this challenge, devices also experience various other accelerations due to we propose a principal component analysis (PCA)based intra-body movements (such as arm and leg movements), scheme to remove the inconsistent accelerations from the which we call inconsistent acceleration.they are usually much observed accelerations.The key observation is that the con- smaller than the consistent acceleration because intra-body sistent acceleration contributes to the observed accelerations movements are usually orders of magnitude smaller than the of all mobile devices attached to the human subject.In other forward movement.When the human subject is moving for- words,the observed acceleration signal of each device is the ward,the approach of simply adding an inertial sensor on the combination of the consistent acceleration signal,which is human chest to directly measure the consistent acceleration from the forwarding movement and the same for all devices, seems to be a feasible solution.however,it cannot work and the inconsistent acceleration signal,which is from the effectively due to the following reasons:First,it still requires intra-body movement from and unique to the device itself. the other mobile devices to synchronize with this specified Furthermore,we observe that the inconsistent acceleration device in the spatial dimension,as the consistent acceleration from the intra-body movement cancels each other out during can only be extracted directly from this device rather than the back and forth movement,and its expected value is close all devices.Usually a magnetometer is essentially required to 0 within a large enough time interval.Thus,the observed for all devices to achieve this synchronization,however,this acceleration signals of multiple devices are strongly correlated, is contradictory to the situation we need to tackle.Second. and therefore we can use PCA-based approach to cancel out even for this specified device.it may lead to inaccuracy the inconsistent factor and extract the consistent factor. in directly measuring the consistent acceleration,since the The second challenge is to address the accumulated errors moving human body may usually introduce some inconsistent in maintaining space synchronization when the human sub- accelerations,more or less,into the measurements on the hor- ject stops moving forward.It is well known that the errors izontal plane.Hence,in this paper,we choose to compute the in gyroscope based oriental tracking accumulate [6],[7]. forwarding direction by extracting the consistent acceleration Furthermore,the errors are further exacerbated by the large from the accelerometer readings.We can treat the inconsistent angular velocities and linear accelerations in human body acceleration as noises and use signal processing techniques motion.Current solutions primarily rely on Kalman filters; to filter them out mostly.Moreover,using a low pass filter however,they only use a single data source from the gyroscope such as a Butterworth filter,the gravitational acceleration to calibrate the accumulated errors [8],which is not sufficient can be extracted from the acceleration measurements in the to further reduce the errors in maintaining space synchro- local coordinate system.Therefore,based on the forwarding nization.To address this challenge,we propose a simple direction and the gravitational direction as reference axes, but effective complementary filter to calibrate the gyroscope the third axis can also be obtained since it is perpendicular to tracking based on the stability of the gravity direction.We the plane defined by these two reference axes.Thus,according propose a rotation model that defines the rotations in theXIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2147 caused by the massive steel embedded in building concrete structures and other metallic objects. Our empirical study further shows that the magnetometer readings are not reli￾able in typical indoor environment. Specifically, for differ￾ent positions in the indoor environment, the angle deviation between the magnetometer measurement and the ground-truth north can be as large as 110◦, the standard deviation of the magnetometer readings can be as large as 26.7◦; for a fixed position in the indoor environment, with the interferences from different sources such as smart phone, earphone, metal plate, and magnet, the angle deviation between the magnetometer measurement and the ground-truth north can be as large as 133.6◦, the standard deviation of the angle deviation can be as large as 86.3◦. B. Proposed Approach In this paper, we propose a scheme called MObile Space Synchronization (MOSS) for devices with two sensors: an accelerometer and a gyroscope, which are available on most mobile devices. Accelerometer readings from multiple mobile devices on a human subject are used to achieve space synchro￾nization when the human subject is moving forward, such as walking and running. Our insight on using accelerometer sen￾sors to achieve space synchronization is that when the human subject moves forward, all attached mobile devices experience the same acceleration along the moving direction of the torso, which we call consistent acceleration. Although these mobile devices also experience various other accelerations due to intra-body movements (such as arm and leg movements), which we call inconsistent acceleration, they are usually much smaller than the consistent acceleration because intra-body movements are usually orders of magnitude smaller than the forward movement. When the human subject is moving for￾ward, the approach of simply adding an inertial sensor on the human chest to directly measure the consistent acceleration seems to be a feasible solution, however, it cannot work effectively due to the following reasons: First, it still requires the other mobile devices to synchronize with this specified device in the spatial dimension, as the consistent acceleration can only be extracted directly from this device rather than all devices. Usually a magnetometer is essentially required for all devices to achieve this synchronization, however, this is contradictory to the situation we need to tackle. Second, even for this specified device, it may lead to inaccuracy in directly measuring the consistent acceleration, since the moving human body may usually introduce some inconsistent accelerations, more or less, into the measurements on the hor￾izontal plane. Hence, in this paper, we choose to compute the forwarding direction by extracting the consistent acceleration from the accelerometer readings. We can treat the inconsistent acceleration as noises and use signal processing techniques to filter them out mostly. Moreover, using a low pass filter such as a Butterworth filter, the gravitational acceleration can be extracted from the acceleration measurements in the local coordinate system. Therefore, based on the forwarding direction and the gravitational direction as reference axes, the third axis can also be obtained since it is perpendicular to the plane defined by these two reference axes. Thus, according to these three axes, we can build the synchronized coordinates with each device as the origins. Furthermore, we use gyroscope readings from multiple mobile devices on a human subject to maintain space syn￾chronization when the human subject stops moving forward, which means that we can no longer obtain the consistent acceleration caused by forward body motion. After the human subject stops moving forward, his body parts may still slightly move or rotate. The gyroscope readings of a mobile device allow us to continuously track the small rotations along the three axes of the device’s local coordinate. We derive a real-time rotation matrix corresponding to the orientation variation by integrating the rotation rates in different axes over time. Leveraging the stability of the gravity direction in the synchronized coordinates, we further calibrate the estimated rotation matrix from gyroscope tracking. Thus, we are able to maintain space synchronization even after the human subject stops moving. C. Technical Challenges and Solutions The first challenge is to extract the consistent acceleration of body movement when the human subject moves forward. This is challenging because both consistent and inconsistent accelerations are mixed together. As the mobile devices are attached to different body locations of the human subject, they perceive rather different accelerations in both the direction and magnitude during human motion. To address this challenge, we propose a principal component analysis (PCA) based scheme to remove the inconsistent accelerations from the observed accelerations. The key observation is that the con￾sistent acceleration contributes to the observed accelerations of all mobile devices attached to the human subject. In other words, the observed acceleration signal of each device is the combination of the consistent acceleration signal, which is from the forwarding movement and the same for all devices, and the inconsistent acceleration signal, which is from the intra-body movement from and unique to the device itself. Furthermore, we observe that the inconsistent acceleration from the intra-body movement cancels each other out during the back and forth movement, and its expected value is close to 0 within a large enough time interval. Thus, the observed acceleration signals of multiple devices are strongly correlated, and therefore we can use PCA-based approach to cancel out the inconsistent factor and extract the consistent factor. The second challenge is to address the accumulated errors in maintaining space synchronization when the human sub￾ject stops moving forward. It is well known that the errors in gyroscope based oriental tracking accumulate [6], [7]. Furthermore, the errors are further exacerbated by the large angular velocities and linear accelerations in human body motion. Current solutions primarily rely on Kalman filters; however, they only use a single data source from the gyroscope to calibrate the accumulated errors [8], which is not sufficient to further reduce the errors in maintaining space synchro￾nization. To address this challenge, we propose a simple but effective complementary filter to calibrate the gyroscope tracking based on the stability of the gravity direction. We propose a rotation model that defines the rotations in the