This article has been accepted for publication in a future issue of this journal,but has not been fully edited.Content may change prior to final publication.Citation information:DOI 10.1109/TMC.2019.2961313.IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING,2019 9 original moving path of the camera which involves the unexpected jitters.Then,to smooth the unexpected jitters moothed Trajectory from the original camera motion,which is usually existing in high frequency band,we apply the low-pass filter on the sequence of [Rto.t}and [Tto.t},respectively,to obtain the smoothed camera motion. To smooth the unexpected jitters in the translation,it is known that the translation vector {Tto.t}has three degrees 02 of freedom,we can apply the low-pass filter on each of 0-0.8-0.6 04y02 the three dimensions directly.Without loss of generality,we Fig.8.The original moving path vs smoothed moving path describe our solution in regard to the z-dimension of the translation vector [Ttot),i.e.,[T).Specifically,we apply original moving path and the positions in the smoothed a weighted moving average filter on the sequence of [T moving path. using a sliding window.To calculate the smoothed trans- 6.3.3 Frame Warping for Video Stabilization lation,we provide different weights at different positions of the sliding window.Here,we use a Gaussian function to During a video shoot,if we do not introduce any stabiliza- calculate the weights,so that for the time t,the weight given tion operation,each image frame generated from the camera to the close positions is higher than those distant positions view is called original frame.However,if we introduce the Assume the length of the sliding window is n,then at the estimated camera motion to calibrate the original image time ti,the smoothed translation Tto.t can be frame,we can replace the original frame with a new image frame,which is called stabilized frame.At first,we transform 、一To,4-1 (20) the pixels Pit corresponding to the feature points in the 1西 original frame to the pixelsPin the stabilized frame.For where wj is the weight at the position j.In this way,we are the feature point Pt in the original frame,the coordinate able to smooth the translation-based jitters of the camera. of its corresponding 3D point at time to,i.e.,Pi.to,can be To smooth the unexpected jitters in the rotation,it is calculated based on the original camera motion.With the known that the 3 x 3 rotation matrix Rto.t involves 9 param- known coordinate Pi.to at time to,and the known smoothed eters.As the camera coordinate system is rotating,we obtain camera motion(Rto.t,Tto.)at the time t by referring to the 9 streams of these parameters over time.However,these pa- status at the time to,we can transform Pi.to to Pi.t,then to rameters are not mutually independent to each other,since Pbased on the camera projection model shown in Eq. the rotation usually has three degrees of freedom in the 3D (23). space.Therefore,in order to smooth the jitters in the rotation (23) measurement,we first transform the 3 x 3 rotation matrix Zi.tPit KPit =K(Rto.Pi,to Tto.t). Rtot into the corresponding Euler angles [tt,to.t,to. Here,Pi.t is the coordinate of P;in the smoothed camera [Ri R12 13 Specifically,suppose Rto.t R21 R22 R23, then,the coordinate system at the time t,Zi.t is the coordinate of Pt R31 R32 R33 in the z-axis,i.e.,the depth value of the pixel PAmong the Euler angles (,t.t,)can be calculated by parameters,Pi.to is obtained through initialization,R.to and Oto.t arctan(Ras Tt.to are obtained through smoothed camera motion,while K 0to.t arctan(- is a known parameter,thus we can calculate Pi.t.After that, Ri2+Ri3 (21) we can obtain Zi.t,which is the coordinate of Pi,t in z-axis. Vto.t=arctan( 21 R11 That is to say,there is only one unknown parameterP Then we use the low pass filter such as the same weighted which can be solved based on Eq.(23).In this way,we can moving average filter to smooth the jitters in the Euler transform each feature point Pt in the original frame to the angles.After that,we further transform the smoothed Euler 3D point Pi.to,and then transform Pi.to to the corresponding angles to the rotation matrix Rto.t in a similar manner as pixel Pt in the stabilized frame. follows: For the other pixels not belonging to feature points, Rto.t=Rio.tRto.Rto.t (22) it is difficult to transform them into the corresponding pixels in the stabilized frame,because of the unknown 3D Here,RRt,Rio.t mean the rotation matrix transformed points corresponding to the pixels.At this time,we combine from the Euler angles oto.t,Oto.t,to.t,respectively.They the original frame and the projected feature points in the also correspond to rotation around x-axis,y-axis,z-axis, stabilized frame to stabilize the video.That is to say,we in- respectively.In this way,we are able to smooth the rotation- troduce a standard texture mapping algorithm according to based jitters of the camera. the warped mesh [15],which proposed content-preserving Fig.8 shows an example of the original moving path and warps to transform each pixel in the original frame to the the smoothed moving path in a global coordinate system, stabilized frame. the moving path involves the rotation and translation of the camera in the 3D space.It is found that the original moving path is full of jitters,while the smoothed moving 6.4 Multi-Thread Optimization for Video Stabilization path is fairly flat and smooth in the contour.The red arrows In order to implement real-time video stabilization,our sys- show the mapping relationship between the positions in the tem needs to output the stabilized frame without noticeable 1536-1233(c)2019 IEEE Personal use is permitted,but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.1536-1233 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMC.2019.2961313, IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING, 2019 9 original moving path of the camera which involves the unexpected jitters. Then, to smooth the unexpected jitters from the original camera motion, which is usually existing in high frequency band, we apply the low-pass filter on the sequence of {Rt0,t} and {Tt0,t}, respectively, to obtain the smoothed camera motion. To smooth the unexpected jitters in the translation, it is known that the translation vector {Tt0,t} has three degrees of freedom, we can apply the low-pass filter on each of the three dimensions directly. Without loss of generality, we describe our solution in regard to the x-dimension of the translation vector {Tt0,t}, i.e., {T x t0,t}. Specifically, we apply a weighted moving average filter on the sequence of {T x t0,t} using a sliding window. To calculate the smoothed trans￾lation, we provide different weights at different positions of the sliding window. Here, we use a Gaussian function to calculate the weights, so that for the time t, the weight given to the close positions is higher than those distant positions. Assume the length of the sliding window is n, then at the time ti , the smoothed translation Tˆ x t0,ti can be Tˆ x t0,ti = Xn j=1 P wj n j=1 wj T x t0,ti−j+1 , (20) where wj is the weight at the position j. In this way, we are able to smooth the translation-based jitters of the camera. To smooth the unexpected jitters in the rotation, it is known that the 3×3 rotation matrix Rt0,t involves 9 param￾eters. As the camera coordinate system is rotating, we obtain 9 streams of these parameters over time. However, these pa￾rameters are not mutually independent to each other, since the rotation usually has three degrees of freedom in the 3D space. Therefore, in order to smooth the jitters in the rotation measurement, we first transform the 3 × 3 rotation matrix Rt0,t into the corresponding Euler angles [φt0,t, θt0,t, ψt0,t]. Specifically, suppose Rt0,t = " R11 R12 R13 R21 R22 R23 R31 R32 R33# , then, the Euler angles (φt0,t, θt0,t, ψt0,t) can be calculated by    φt0,t = arctan( R32 R33 ), θt0,t = arctan(√ −R31 R2 32+R2 33 ), ψt0,t = arctan( R21 R11 ). (21) Then we use the low pass filter such as the same weighted moving average filter to smooth the jitters in the Euler angles. After that, we further transform the smoothed Euler angles to the rotation matrix Rbt0,t in a similar manner as follows: Rbt0,t = R z t0,tR y t0,tR x t0,t. (22) Here, R x t0,t, R y t0,t, R z t0,t mean the rotation matrix transformed from the Euler angles φt0,t, θt0,t, ψt0,t, respectively. They also correspond to rotation around x-axis, y-axis, z-axis, respectively. In this way, we are able to smooth the rotation￾based jitters of the camera. Fig. 8 shows an example of the original moving path and the smoothed moving path in a global coordinate system, the moving path involves the rotation and translation of the camera in the 3D space. It is found that the original moving path is full of jitters, while the smoothed moving path is fairly flat and smooth in the contour. The red arrows show the mapping relationship between the positions in the y z x -1 -1 0 -0.5 1 0.2 0 -0.2 2 -0.4 0 -0.6 -0.8 Original Trajectory with Jitters Smoothed Trajectory y x -1 -1 -0.8 -0.6 0 -0.4 z -0.2 0.1 0 -0.1 -0.2 1 -0.3 -0.4 0 -0.5 -0.6 -0.7 2 Original Trajectory Smoothed Trajectory Mapping relationship between original position and smoothed position Fig. 8. The original moving path vs smoothed moving path original moving path and the positions in the smoothed moving path. 6.3.3 Frame Warping for Video Stabilization During a video shoot, if we do not introduce any stabiliza￾tion operation, each image frame generated from the camera view is called original frame. However, if we introduce the estimated camera motion to calibrate the original image frame, we can replace the original frame with a new image frame, which is called stabilized frame. At first, we transform the pixels P 0 i,t corresponding to the feature points in the original frame to the pixels Pˆ 0 i,t in the stabilized frame. For the feature point P 0 i,t in the original frame, the coordinate of its corresponding 3D point at time t0, i.e., Pi,t0 , can be calculated based on the original camera motion. With the known coordinate Pi,t0 at time t0, and the known smoothed camera motion (Rˆt0,t, Tˆt0,t) at the time t by referring to the status at the time t0, we can transform Pi,t0 to Pˆi,t, then to Pˆ 0 i,t, based on the camera projection model shown in Eq. (23). Zˆ i,t · Pˆ 0 i,t = KPˆi,t = K(Rˆt0,tPi,t0 + Tˆt0,t). (23) Here, Pˆi,t is the coordinate of Pi in the smoothed camera coordinate system at the time t, Zˆ i,t is the coordinate of Pˆi,t in the z-axis, i.e., the depth value of the pixel Pˆ 0 i,t. Among the parameters, Pi,t0 is obtained through initialization, Rˆt,t0 and Tˆt,t0 are obtained through smoothed camera motion, while K is a known parameter, thus we can calculate Pˆi,t. After that, we can obtain Zˆ i,t, which is the coordinate of Pˆi,t in z-axis. That is to say, there is only one unknown parameter Pˆ 0 i,t, which can be solved based on Eq. (23). In this way, we can transform each feature point P 0 i,t in the original frame to the 3D point Pi,t0 , and then transform Pi,t0 to the corresponding pixel Pˆ 0 i,t in the stabilized frame. For the other pixels not belonging to feature points, it is difficult to transform them into the corresponding pixels in the stabilized frame, because of the unknown 3D points corresponding to the pixels. At this time, we combine the original frame and the projected feature points in the stabilized frame to stabilize the video. That is to say, we in￾troduce a standard texture mapping algorithm according to the warped mesh [15], which proposed content-preserving warps to transform each pixel in the original frame to the stabilized frame. 6.4 Multi-Thread Optimization for Video Stabilization In order to implement real-time video stabilization, our sys￾tem needs to output the stabilized frame without noticeable