XIE et al:SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2155 simple and regular.However,in more general activities,e.g., the activities for VR applications,in which the motions can lead to sudden,rapid and irregular acceleration and orientation changes,MOSS may not achieve ideal performance for the following reasons:1)In sudden activities,the inconsistent accelerations can be comparable to the consistent acceleration in the magnitude of the movement,such that the consistent 400 acceleration cannot be effectively extracted.Besides,as afore- Sampling Point (50Hz) mentioned,the time window for the consistent acceleration can Fig.13. Inertial readings with/without Kalman filter. be too small,this further increases the difficulty to extract the within a time window,respectively.Take the acceleration in consistent acceleration.2)Sudden activity may lead to large r-axis as an example,we use 1to denote the estimated errors in the inertial measurements including the accelerometer acceleration at time t-1.Then,we use Kalman filter to measurements and gyroscope measurements.The errors in estimate the true acceleration at time tk,while combining the accelerometer measurements further lead to difficulty in the last estimated acceleration k1 at time t-1 and the achieving space synchronization,whereas the errors in the measurement z from the accelerometer at time t.Kalman gyroscope measurements further lead to difficulty in maintain- filter is a recursive estimator,which contains a prediction phase ing space synchronization.Therefore,MOSS is more or less and an update phase in each recursive process. vulnerable to the sudden activities due to the above reasons. In the prediction phase,we use k-1lk1 to predict the next acceleration as klk-1,i.e.,Tklk-1 Tk-11k-1.The IX.PERFORMANCE EVALUATION estimate covariance Pk-1=P-1k-1+Q,where P-1k-1 A.Implementation and Setup means the estimate covariance in the last update phase,while O means the covariance of the process noise in the prediction Hardware:We implemented MOSS on 1 smart glasses phase.Here,the process noise is assumed as Gaussian white (Google Glass 2)and 5 smart phones(SAMSUNG Galaxy S5). They were placed on a human subject as shown in Fig.2. noise N(uQ,Q)(Q =1.0004 in our implementation). In the update phase,we combine the predicted acceleration These devices were connected to an Android smartphone (SAMSUNG Galaxy S5)via the Bluetooth 4.0 interface, k-1 and the measurement zk to update the estimate of the i.e.,S2 in Fig.2.The Android smartphone S2 continuously acceleration klk at time tk,i.e.,klk klk-1+Kk(zk- collected the accelerometer and gyroscope measurements from k-1).Here,K&means the Kalman gain at time tk,and it these devices,and processed these measurements. can be calculated as follows:Kk=Pk-1(Pk-1+R)-1, where R means the covariance of the measurement noise Setup:We let 5 volunteers move along three different in the update phase,and the measurement noise is assumed straight paths in outdoor environment.They moved along the paths in three different modes:1)walk:the human subject as Gaussian white noise N(uR,R)(R 0.95976 in our implementation).In the update phase,the estimate covariance walked in normal speed (0.5~1 m/s)with small-amplitudes; 2)run:the human subject ran in fairly fast speed(2~3m/s) Pk=(1-Kk)P-1.Therefore,the optimal estimation of with large-amplitudes;3)jump:the human subject jumped acceleration in r-axis at time tk is klk.Similarly,we can obtain the optimal estimation of the acceleration in y-axis. forward with large-amplitudes.For each path,we collected 20 traces for each moving mode.In each trace,we col- and z-axis at time t as ylk and k respectively.After that, lected the inertial measurements from the accelerometers. we average the values ofwithin a time window, gyroscopes,and magnetometers of all devices for a maximum respectively,to infer the forwarding direction.The solution is effective when the body frames of the devices do not change time interval of 30 seconds.We collected a total of 120 traces too much during the moving process. of inertial measurements.To validate the performance against the ground truth,we extracted the magnetic direction and 2)RefMeanAcc:it estimates the forwarding direction by averaging the acceleration measurements within a time win- gravity direction from the inertial measurement,and used them dow respectively in three axes of the reference local coordi- to build an earth coordinate system as the reference coordinate nate.The main difference between MeanAcc and RefMeanAcc system E. is that RefMeanAcc transforms all the acceleration from the Metrics:We used two metrics to evaluate MOSS:(1)Angle body frame to the reference local coordinate.After that, Accuracy:the angle deviation between the estimated forward- ing direction and the ground truth,(2)Coordinate Accuracy: RefMeanAcc calculates the acceleration in each axis based on Kalman filter [31]in the same way described in MeanAcc. the correlations between the measurements from the synchro- The solution is effective even when the body frames of the nized coordinate and ground truth. devices keep changing,using the gyroscope tracking Experimental results show that Kalman filter can effectively B.Forwarding Direction Estimation remove the outliers or mitigate the jitters of the raw iner- We compared our solution with two heuristic solutions using tial readings.Fig.13 shows the corresponding inertial readings Kalman filter as baseline solutions.i.e.. with or without the Kalman filter.Note that in comparison 1)MeanAcc:it uses Kalman filter [31]to filter the inertial to the raw inertial measurement,the inertial readings after readings first,and then estimates the forwarding direction by the Kalman filter are more smooth and have less jitters.It averaging the acceleration in three axes of the body frame implies that,for the situations where the human motions leadXIE et al.: SYNCHRONIZE INERTIAL READINGS FROM MULTIPLE MOBILE DEVICES IN SPATIAL DIMENSION 2155 simple and regular. However, in more general activities, e.g., the activities for VR applications, in which the motions can lead to sudden, rapid and irregular acceleration and orientation changes, MOSS may not achieve ideal performance for the following reasons: 1) In sudden activities, the inconsistent accelerations can be comparable to the consistent acceleration in the magnitude of the movement, such that the consistent acceleration cannot be effectively extracted. Besides, as afore￾mentioned, the time window for the consistent acceleration can be too small, this further increases the difficulty to extract the consistent acceleration. 2) Sudden activity may lead to large errors in the inertial measurements including the accelerometer measurements and gyroscope measurements. The errors in the accelerometer measurements further lead to difficulty in achieving space synchronization, whereas the errors in the gyroscope measurements further lead to difficulty in maintain￾ing space synchronization. Therefore, MOSS is more or less vulnerable to the sudden activities due to the above reasons. IX. PERFORMANCE EVALUATION A. Implementation and Setup Hardware: We implemented MOSS on 1 smart glasses (Google Glass 2) and 5 smart phones (SAMSUNG Galaxy S5). They were placed on a human subject as shown in Fig. 2. These devices were connected to an Android smartphone (SAMSUNG Galaxy S5) via the Bluetooth 4.0 interface, i.e., S2 in Fig. 2. The Android smartphone S2 continuously collected the accelerometer and gyroscope measurements from these devices, and processed these measurements. Setup: We let 5 volunteers move along three different straight paths in outdoor environment. They moved along the paths in three different modes: 1) walk: the human subject walked in normal speed (0.5∼1 m/s) with small-amplitudes; 2) run: the human subject ran in fairly fast speed (2∼3m/s) with large-amplitudes; 3) jump: the human subject jumped forward with large-amplitudes. For each path, we collected 20 traces for each moving mode. In each trace, we col￾lected the inertial measurements from the accelerometers, gyroscopes, and magnetometers of all devices for a maximum time interval of 30 seconds. We collected a total of 120 traces of inertial measurements. To validate the performance against the ground truth, we extracted the magnetic direction and gravity direction from the inertial measurement, and used them to build an earth coordinate system as the reference coordinate system E. Metrics: We used two metrics to evaluate MOSS: (1) Angle Accuracy: the angle deviation between the estimated forward￾ing direction and the ground truth, (2) Coordinate Accuracy: the correlations between the measurements from the synchro￾nized coordinate and ground truth. B. Forwarding Direction Estimation We compared our solution with two heuristic solutions using Kalman filter as baseline solutions, i.e., 1) MeanAcc: it uses Kalman filter [31] to filter the inertial readings first, and then estimates the forwarding direction by averaging the acceleration in three axes of the body frame Fig. 13. Inertial readings with/without Kalman filter. within a time window, respectively. Take the acceleration in x-axis as an example, we use xk−1|k−1 to denote the estimated acceleration at time tk−1. Then, we use Kalman filter to estimate the true acceleration xk|k at time tk, while combining the last estimated acceleration xk−1|k−1 at time tk−1 and the measurement zk from the accelerometer at time tk. Kalman filter is a recursive estimator, which contains a prediction phase and an update phase in each recursive process. In the prediction phase, we use xk−1|k−1 to predict the next acceleration as xk|k−1, i.e., xk|k−1 = xk−1|k−1. The estimate covariance Pk|k−1 = Pk−1|k−1 +Q, where Pk−1|k−1 means the estimate covariance in the last update phase, while Q means the covariance of the process noise in the prediction phase. Here, the process noise is assumed as Gaussian white noise N(μQ, Q)(Q = 1.0004 in our implementation). In the update phase, we combine the predicted acceleration xk|k−1 and the measurement zk to update the estimate of the acceleration xk|k at time tk, i.e., xk|k = xk|k−1 + Kk(zk − xk|k−1). Here, Kk means the Kalman gain at time tk, and it can be calculated as follows: Kk = Pk|k−1(Pk|k−1 + R)−1, where R means the covariance of the measurement noise in the update phase, and the measurement noise is assumed as Gaussian white noise N(μR, R) (R = 0.95976 in our implementation). In the update phase, the estimate covariance Pk|k = (1 − Kk)Pk|k−1. Therefore, the optimal estimation of acceleration in x-axis at time tk is xk|k. Similarly, we can obtain the optimal estimation of the acceleration in y-axis, and z-axis at time tk as yk|k and zk|k, respectively. After that, we average the values of xk|k, yk|k, zk|k within a time window, respectively, to infer the forwarding direction. The solution is effective when the body frames of the devices do not change too much during the moving process. 2) RefMeanAcc: it estimates the forwarding direction by averaging the acceleration measurements within a time win￾dow respectively in three axes of the reference local coordi￾nate. The main difference between MeanAcc and RefMeanAcc is that RefMeanAcc transforms all the acceleration from the body frame to the reference local coordinate. After that, RefMeanAcc calculates the acceleration in each axis based on Kalman filter [31] in the same way described in MeanAcc. The solution is effective even when the body frames of the devices keep changing, using the gyroscope tracking. Experimental results show that Kalman filter can effectively remove the outliers or mitigate the jitters of the raw iner￾tial readings. Fig. 13 shows the corresponding inertial readings with or without the Kalman filter. Note that in comparison to the raw inertial measurement, the inertial readings after the Kalman filter are more smooth and have less jitters. It implies that, for the situations where the human motions lead