IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 12 Sensors SpeedTalker Acoustic Signal Processing Speed Microphones Low-Pass Time Delay Segmentation Time Delay Estimation Filtering Calculation Curve Smoothing Trajectory IMU Rotation Calculation Estimation Computer Vision based Processing Speed Camera Automobile Automobile Type Wheel Hub Distance Calculation Extracting Recognition Extracting Estimation Fig.14:System Design. modified to Adm or -Adm.Then,the time delays in Major The second step is to recognize the type of the auto- Detection Region should not change rapidly.That means mobile so as to get the type of the wheel hub.Each type the difference between the adjacent time delays should not of automobile has its well-matched wheel hub size[28]. exceed a threshold.We can see from figure 6a that the largest If we recognize the type of the automobile,we can get gradient of the curve is at the time when Ad=0.Without the real diameter H of the wheel hub.There are several loss of generality,we suppose the vertical distance between computer vision tools based on machine learning can recog- the automobile and the mobile phone L is 5 m,the distance nize the type,such as DeepVision[29],OrpixVision[30]and between the two microphones 2l is 0.2 m.If the difference BaiduAI[31].The method using computer vision gives APIs between adjacent time delays is 3,the speed of the auto- for our system to recognize the type of the automobile as mobile is 75 m/s(270 km/h),which is rarely seen in daily figure 16b shows. life.As a result,the difference between adjacent time delays After we get the real parameter H of the wheel hub,we should less than 3 in ideal situations.We relax the threshold need to calculate the pixel parameter h of the wheel hub to 5.So we can exclude extremely abnormal time delays in the image.The shape of the wheel hub is a circle.Due which may influence the traditional smoothing method and to the movement of the automobile,the wheel hubs in the retain other time delays.The value of extremely abnormal image are not standard circles.However,the movement of time delays can be modified to that of its neighbors. the automobile is in the horizontal direction,which has no The second step is to smooth the sequence with tradi- effect on the diameter in the upright direction.As a result, tional smoothing methods,e.g.,Gaussian smoothing.Then we focus on the wheel hub in the upright direction.The we will get the smoothed time delay curve as figure 6b method to detect the rough circle is using Hough transform. shows. Hough transform can be utilized to arbitrary shapes[32]. First we detect the edge of the automobile with Canny edge 4.3 Computer Vision Based Processing detection algorithm[33].To achieve better performance,we After we smooth the delay curve and get the Major Detec- cut out the lower right corner of the automobile extracted tion Region,we may get a series of candidate trajectories from the figure since the wheel hub of the automobile are with the same slope m.However,the other parameter b mostly appear at the bottom of the image.We may find the of the trajectory cannot be estimated by acoustic signals. wheel hub by Hough transform.The vertical distance L can So we use the image of the automobile to complete the be calculated through equation(12). estimation of the trajectory.From equation 12 we can see With yolo we also get the position of the automobile in that we need to calculate the pixel diameter h of the wheel the image,with equation (13),the offset angle can also be hub and the real diameter H of the wheel hub.The key point calculated. of this section is to extract the wheel hub information in the image and in reality.Figure 16 shows the process of image processing,including Automobile Extraction,Automoblie Type 4.4 Jitters Removing Recognition,Hough Transform and Diameter Estimation. Since we get the approach of time delay modification in Sec- First we need to extract the automobile in the image. tion 3.2.5,the following part of this section is to calculate the The image we get from the camera contains too many rotation angle around x-axis.We define the moment when objects,which makes it difficult to recognize the type of the we start to record the sound of the automobile as 70.The co- automobile.Besides,too many pixels in the image increase ordinate system at To is denoted as CTo.We need to estimate the complexity of image processing.We just want to pay the speed of the automobile in a stable reference system. attention to the automobile itself.As a result,we utilize As a result,we transform the time delay Ad calculated yolo[27],which applies a single neural network to the full by cross-correlation in altered coordinate system Cr into image.The network divides the image into regions and pre- the time delay Ad'in Cro.We have already analyze how dicts bounding boxes and probabilities for each region.With different translations and rotations will influence the time Yolo,we can extract the automobile and get the position of delay.The influence of the jitters can be resolved into these the automobile in the image.Some empirical approaches translations and rotations.And we just need to achieve the are utilized to solve the problem.The pixel locations of the rotation around x-axis.We utilize rotation vector in android car should be a rectangle whose length-width ratio is more system[34]to calculate the rotation.The rotation vector rep- than 2.And the automobile should not be static in different resents the orientation of the device as a combination of an frames. angle and an axis,in which the device has rotated through Authorized licensed use limited to:Nanjing University.Downloaded on December 24,2020 at 09:11:21 UTC from IEEE Xplore.Restrictions applyIEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 12 Microphones IMU Camera Sensors SpeedTalker Low-Pass Filtering Segmentation Time Delay Calculation Time Delay Curve Smoothing Acoustic Signal Processing Speed Estimation Trajectory Estimation Speed Calculation Automobile Extracting Automobile Type Recognition Wheel Hub Extracting Distance Estimation Computer Vision based Processing Rotation Calculation Fig. 14: System Design. modified to ∆dm or −∆dm. Then, the time delays in Major Detection Region should not change rapidly. That means the difference between the adjacent time delays should not exceed a threshold. We can see from figure 6a that the largest gradient of the curve is at the time when ∆d = 0. Without loss of generality, we suppose the vertical distance between the automobile and the mobile phone L is 5 m, the distance between the two microphones 2l is 0.2 m. If the difference between adjacent time delays is 3, the speed of the auto￾mobile is 75 m/s(270 km/h), which is rarely seen in daily life. As a result, the difference between adjacent time delays should less than 3 in ideal situations. We relax the threshold to 5. So we can exclude extremely abnormal time delays which may influence the traditional smoothing method and retain other time delays. The value of extremely abnormal time delays can be modified to that of its neighbors. The second step is to smooth the sequence with tradi￾tional smoothing methods, e.g., Gaussian smoothing. Then we will get the smoothed time delay curve as figure 6b shows. 4.3 Computer Vision Based Processing After we smooth the delay curve and get the Major Detec￾tion Region, we may get a series of candidate trajectories with the same slope m. However, the other parameter b of the trajectory cannot be estimated by acoustic signals. So we use the image of the automobile to complete the estimation of the trajectory. From equation 12 we can see that we need to calculate the pixel diameter h of the wheel hub and the real diameter H of the wheel hub. The key point of this section is to extract the wheel hub information in the image and in reality. Figure 16 shows the process of image processing, including Automobile Extraction, Automoblie Type Recognition, Hough Transform and Diameter Estimation. First we need to extract the automobile in the image. The image we get from the camera contains too many objects, which makes it difficult to recognize the type of the automobile. Besides, too many pixels in the image increase the complexity of image processing. We just want to pay attention to the automobile itself. As a result, we utilize yolo[27], which applies a single neural network to the full image. The network divides the image into regions and pre￾dicts bounding boxes and probabilities for each region. With Yolo, we can extract the automobile and get the position of the automobile in the image. Some empirical approaches are utilized to solve the problem. The pixel locations of the car should be a rectangle whose length-width ratio is more than 2. And the automobile should not be static in different frames. The second step is to recognize the type of the auto￾mobile so as to get the type of the wheel hub. Each type of automobile has its well-matched wheel hub size[28]. If we recognize the type of the automobile, we can get the real diameter H of the wheel hub. There are several computer vision tools based on machine learning can recog￾nize the type, such as DeepVision[29], OrpixVision[30] and BaiduAI[31]. The method using computer vision gives APIs for our system to recognize the type of the automobile as figure 16b shows. After we get the real parameter H of the wheel hub, we need to calculate the pixel parameter h of the wheel hub in the image. The shape of the wheel hub is a circle. Due to the movement of the automobile, the wheel hubs in the image are not standard circles. However, the movement of the automobile is in the horizontal direction, which has no effect on the diameter in the upright direction. As a result, we focus on the wheel hub in the upright direction. The method to detect the rough circle is using Hough transform. Hough transform can be utilized to arbitrary shapes[32]. First we detect the edge of the automobile with Canny edge detection algorithm[33]. To achieve better performance, we cut out the lower right corner of the automobile extracted from the figure since the wheel hub of the automobile are mostly appear at the bottom of the image. We may find the wheel hub by Hough transform. The vertical distance L can be calculated through equation (12). With yolo we also get the position of the automobile in the image, with equation (13), the offset angle can also be calculated. 4.4 Jitters Removing Since we get the approach of time delay modification in Sec￾tion 3.2.5, the following part of this section is to calculate the rotation angle around x-axis. We define the moment when we start to record the sound of the automobile as T0. The co￾ordinate system at T0 is denoted as CT0 . We need to estimate the speed of the automobile in a stable reference system. As a result, we transform the time delay ∆d calculated by cross-correlation in altered coordinate system CTn into the time delay ∆d 0 in CT0 . We have already analyze how different translations and rotations will influence the time delay. The influence of the jitters can be resolved into these translations and rotations. And we just need to achieve the rotation around x-axis. We utilize rotation vector in android system[34] to calculate the rotation. The rotation vector rep￾resents the orientation of the device as a combination of an angle and an axis, in which the device has rotated through Authorized licensed use limited to: Nanjing University. Downloaded on December 24,2020 at 09:11:21 UTC from IEEE Xplore. Restrictions apply