This article has been accepted for publication in a future issue of this journal,but has not been fully edited.Content may change prior to final publication.Citation information:DOI 10.1109/TMC.2019.2961313.IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING,2019 十4 100150200250 40 150 n 45 90 10 10-15 25-30 40-45 target point cm) target p Angle of rotation(deg) Dista of translation(cm) (a)The jitter of pixels due to(b)The jitter of pixels due to (c)The rotation measurement error (d)The translation measurement rotation-based jitter,60 =100 translation-based jitter,ot 5cm from gyroscope error from accelerometer Fig.3.The experiment results of the empirical study. 4.1 Observations from 130 pixels to 43 pixels Observation 1.When the camera is subject to the same rotation- Observation 3.When the mobile device is rotating,the gyro- scope is able to accurately measure the rotation in low,medium based jitters,the stationary target points with closer distance and high speed mode.To evaluate the average measurement to the image plane are suffering from stronger pixel jitters in the image plane.To evaluate how the rotation-based jitters errors in measuring the rotation with the gyroscope,without affect the pixel jitters in the image plane,we deployed the loss of generality,we rotated the mobile device around the stationary target points in a line parallel to the optical axis z-axis of the local coordinate system with the angle of 45 of camera and with different distances to the image plane. 90°andl80°，respectively.Besides,for each rotation angle,, Without loss of generality,we performed the rotation-based we evaluate the measurement errors with the low speed (10°/s),medium speed(40°/s)and high speed(100°/s) jitters around the y-axis of the camera coordinate system, which leads to the coordinate change of the projection in mode,respectively.Specifically,the measurement errors are x-axis.The maximum rotation angle 60 is set to 10 degrees calculated by comparing the gyroscope measurement with by default.Then,we measured the pixel jitter for the target the ground truth.According to the experiment results in points with different depths,i.e.,coordinate difference in Fig.3(c),we found that,as the rotation angle increases from pixels between the projections before and after the rotation- 45°to180°，the measurement error is slightly increasing, which is always less than 2 in all cases. based jitter.As shown in Fig.3(a),we use the pinhole camera model to predict the pixel jitter of an object at a given Observation 4.When the mobile device is moving back and forth,the accelerometer usually fails to accurately measure the distance,and plot it as the curve shown in green color,then we plot the corresponding experiment results for pixel jitter translation in low,medium and high speed mode.To evaluate of an object at a given distance.The comparison between the average measurement errors in measuring the trans- the theoretical results and the experiment results shows that lation with the accelerometer,without loss of generality, the observations from the experiments are consistent with we move the mobile device back and forth in the range the theoretical hypothesis from the pinhole camera model. of [-5cm,+5cm]along the z-axis of the local coordinate According to the experiment results,we found that as the system,by varying the overall distance from 10~15cm to depth of the target point increases from 10cm to 50cm,the 40~45cm,respectively.Besides,for each moving distance, pixel jitter decreases rapidly from 314 pixels to 235 pixels. we evaluate the measurement errors with the low speed Then,as the depth further increases from 50cm to 150cm (3cm/s),medium speed(30cm/s)and high speed(100cm/s) the pixel jitter decreases very slowly from 235 pixels to 230 mode,respectively.Specifically,the measurement errors are pixels. calculated by comparing the accelerometer measurement with the ground truth.As shown in Fig.3(d),we found Observation 2.When the camera is subiect to the same translation-based jitters,the stationary target points with closer that,for all three speed modes,as the moving distance distance to the image plane are suffering from stronger pixel increases from 10~15cm to 40~45cm,the corresponding jitters in the image plane.To evaluate how the translation- measurement errors are linearly increasing.Nevertheless, the measurement errors of all speed modes with all moving based jitters affect the pixel jitters in the image plane of the camera,we deployed the target points in the optical axis of distances are all greater than 10cm.Since the actual trans- the camera and with different distances to the image plane. lation ranges in [-5cm,+5cm],and the maximum moving Without loss of generality,we performed the translation- distance is less than 45cm,thus the average measurement error (whether displacement error or distance error)greater based jitters along the x-axis of the camera coordinate system,the maximum displacement 6t is set to 5cm by than 10cm is not acceptable at all. default.Then,we also measured the pixel jitter for the target 4.2 Summary points with different depths.As shown in Fig.3(b),we use the pinhole camera model to predict the pixel jitter of an Both the rotation-based jitters and the translation-based object at a given distance,and plot it as the curve shown jitters cause non-negligible pixel jitters in the image plane in green color,then we plot the corresponding experiment during video shoot.With the inertial measurement units, results for pixel jitter of an object at a given distance.We usually the rofation can be accurately measured by the found that as the depth of the target point increases from gyroscope,whereas the translation fails to be accurately 10cm to 50cm,the pixel jitter decreases rapidly from 650 measured by the accelerometer.Therefore,it is essential pixels to 130 pixels.Then,as the depth further increases to estimate the translation in an accurate and lightweight from 50cm to 150cm,the pixel jitter decreases very slowly manner,such that the video stabilization can be effectively performed. 1536-1233(c)2019 IEEE Personal use is permitted,but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.1536-1233 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMC.2019.2961313, IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING, 2019 4 0 50 100 150 200 250 300 Distance of target points (cm) 220 240 260 280 300 320 Pixel jitters (pixel) (a) The jitter of pixels due to rotation-based jitter, δθ = 10◦ 0 50 100 150 200 250 300 Distance of target points (cm) 0 200 400 600 800 Pixel jitters (pixel) (b) The jitter of pixels due to translation-based jitter, δt = 5cm 45 90 180 Angle of rotation(deg) 0 1 2 3 Measurement error(deg) Slow speed Medium speed Fast speed (c) The rotation measurement error from gyroscope 10-15 25-30 40-45 Distance of translation(cm) 0 10 20 30 40 50 Translation error(cm) Slow speed Medium speed Fast speed (d) The translation measurement error from accelerometer Fig. 3. The experiment results of the empirical study. 4.1 Observations Observation 1. When the camera is subject to the same rotation￾based jitters, the stationary target points with closer distance to the image plane are suffering from stronger pixel jitters in the image plane. To evaluate how the rotation-based jitters affect the pixel jitters in the image plane, we deployed the stationary target points in a line parallel to the optical axis of camera and with different distances to the image plane. Without loss of generality, we performed the rotation-based jitters around the y-axis of the camera coordinate system, which leads to the coordinate change of the projection in x-axis. The maximum rotation angle δθ is set to 10 degrees by default. Then, we measured the pixel jitter for the target points with different depths, i.e., coordinate difference in pixels between the projections before and after the rotation￾based jitter. As shown in Fig.3(a), we use the pinhole camera model to predict the pixel jitter of an object at a given distance, and plot it as the curve shown in green color, then we plot the corresponding experiment results for pixel jitter of an object at a given distance. The comparison between the theoretical results and the experiment results shows that the observations from the experiments are consistent with the theoretical hypothesis from the pinhole camera model. According to the experiment results, we found that as the depth of the target point increases from 10cm to 50cm, the pixel jitter decreases rapidly from 314 pixels to 235 pixels. Then, as the depth further increases from 50cm to 150cm, the pixel jitter decreases very slowly from 235 pixels to 230 pixels. Observation 2. When the camera is subject to the same translation-based jitters, the stationary target points with closer distance to the image plane are suffering from stronger pixel jitters in the image plane. To evaluate how the translation￾based jitters affect the pixel jitters in the image plane of the camera, we deployed the target points in the optical axis of the camera and with different distances to the image plane. Without loss of generality, we performed the translation￾based jitters along the x-axis of the camera coordinate system, the maximum displacement δt is set to 5cm by default. Then, we also measured the pixel jitter for the target points with different depths. As shown in Fig.3(b), we use the pinhole camera model to predict the pixel jitter of an object at a given distance, and plot it as the curve shown in green color, then we plot the corresponding experiment results for pixel jitter of an object at a given distance. We found that as the depth of the target point increases from 10cm to 50cm, the pixel jitter decreases rapidly from 650 pixels to 130 pixels. Then, as the depth further increases from 50cm to 150cm, the pixel jitter decreases very slowly from 130 pixels to 43 pixels. Observation 3. When the mobile device is rotating, the gyro￾scope is able to accurately measure the rotation in low, medium and high speed mode. To evaluate the average measurement errors in measuring the rotation with the gyroscope, without loss of generality, we rotated the mobile device around the z-axis of the local coordinate system with the angle of 45◦ , 90◦ and 180◦ , respectively. Besides, for each rotation angle, we evaluate the measurement errors with the low speed (10◦/s), medium speed (40◦/s) and high speed (100◦/s) mode, respectively. Specifically, the measurement errors are calculated by comparing the gyroscope measurement with the ground truth. According to the experiment results in Fig.3(c), we found that, as the rotation angle increases from 45◦ to 180◦ , the measurement error is slightly increasing, which is always less than 2◦ in all cases. Observation 4. When the mobile device is moving back and forth, the accelerometer usually fails to accurately measure the translation in low, medium and high speed mode. To evaluate the average measurement errors in measuring the trans￾lation with the accelerometer, without loss of generality, we move the mobile device back and forth in the range of [-5cm, +5cm] along the z-axis of the local coordinate system, by varying the overall distance from 10∼15cm to 40∼45cm, respectively. Besides, for each moving distance, we evaluate the measurement errors with the low speed (3cm/s), medium speed (30cm/s) and high speed (100cm/s) mode, respectively. Specifically, the measurement errors are calculated by comparing the accelerometer measurement with the ground truth. As shown in Fig.3(d), we found that, for all three speed modes, as the moving distance increases from 10∼15cm to 40∼45cm, the corresponding measurement errors are linearly increasing. Nevertheless, the measurement errors of all speed modes with all moving distances are all greater than 10cm. Since the actual trans￾lation ranges in [-5cm, +5cm], and the maximum moving distance is less than 45cm, thus the average measurement error (whether displacement error or distance error) greater than 10cm is not acceptable at all. 4.2 Summary Both the rotation-based jitters and the translation-based jitters cause non-negligible pixel jitters in the image plane during video shoot. With the inertial measurement units, usually the rotation can be accurately measured by the gyroscope, whereas the translation fails to be accurately measured by the accelerometer. Therefore, it is essential to estimate the translation in an accurate and lightweight manner, such that the video stabilization can be effectively performed