This article has been accepted for publication in a future issue of this journal,but has not been fully edited.Content may change prior to final publication.Citation information:DOI 10.1109/TMC.2020.3034354.IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 11 20 Calibrated by multiple peaks6.74 -20 10 20 25 15.5 16 16.5 17 17.5 18 500 -250 250 500 Time(s) Time(s) Time Delay(sample) (a)Time delays of microphones with 12 automo-(b)Time delays when 7th and 8th automobiles (c)Cross-correlation curve when t=16.74s. biles. pass by. Fig.13:Processing of acoustic signals of multiple automobiles. in the correlation figures to recover the time delay curve. curve can be smoothed with Gaussian smoothing.Computer Then we focus on the correlation between the corresponding Vision Based Processing first extracts the automobile in the segments at time 16.74s.In figure 13a,we have time delay image with yolo27].The automobile extracted from the △d=10 around t=16.4 s and time delay△d=-5 image will be put into a deep learning network to recognize around t=16.6s.According to these existing points,we the type of the automobile and know the diameter of its can estimate the range of the time delay when t 16.74s. wheel hub in reality.Then we extract the wheel hub through Figure 13c shows the cross-correlation of the corresponding Hough Transform,get the pixel diameter of the wheel hub segments.Several peaks appear in the figure.We calculate in the image and estimate the vertical distance between the top three peaks in correlation figure.We can see the second automobile and the mobile phone.At last,Speed Extraction highest peak is located on time delay-15 and is suitable recovers the trajectory of the automobile and estimates the for the time delay curve.That means the highest peak at speed of the automobile. Ad 19 is caused by automobile 8 and the second highest peak at Ad =-15 is caused by automobile 7.At last 4.2 Acoustic Signal Processing if no time delay satisfy the range,we will abandon the corresponding segments The acoustic signals we collect usually contain many noises. Case 3:Only one of the automobiles becomes Major Detection We can see from figure 2b that the main energy of the Object when multiple automobiles ao through the Major Detection sound made by the automobile is distributed in the low- Region. frequency area.To make the time delay calculated through The situation is shown in figure 11c.In this case the cross-correlation more accurate,we have let the sound of line of sight(LOS)path between the automobiles overlap automobile dominate the acoustic signal.As a result,we use within the major detection region.That means the delay a low-pass filter to remove the high-frequency noises.Dif- curve of these automobiles can not be separated from each ferent types of automobiles have different noise frequency other.The speed of the automobiles cannot be estimated in distribution,but we know that the automobile noise is this case.In this case,the sound collected by microphones mainly distributed in the frequencies below 4 kHz.So we is too complicated to analyze.As a result,we conduct use a low-pass filter with the cutoff frequency of 4 kHz. experiments on multiple automobiles and the result shows Next,we need to split the acoustic signals into seg- that if SpeedTalker is not utilized at morning or evening ments sorted by time.As mentioned before,the size of the peak,the detection success rate is about 92%.In figure 13a, segments we design is fs/100.After getting two series of the curve in red of the 11th and 12th automobiles cannot be audio segments,we calculate the cross-correlation between distinguished.The two delay curve highly overlap and the the segment pairs to make out the time delay between the two acoustic signals of the automobiles are severely affected corresponding segments.Then we get a series of time delays by each other.Only the speed of the automobile which can with timestamp,which can be used to draw the time delay be detected by the cameras can be estimated. curve.The illustration of the process is shown in figure 15. At last,we need to smooth the time delay curve.In Section 3.2.2,we propose two constraints to filter the time 4 SYSTEM DESIGN delays.We can divide the whole curve into two parts: Minor Detection Region and Major Detection Region.Minor 4.1 System Overview Detection Region refers to the region where the time delay The system architecture is shown in figure 14.There are remains unchanged,with Ad Adm.The definition of Ma- three main components in SpeedTalker,i.e.,Acoustic Sig- jor Detection Region is in Section 3.2.2.When the acoustic nal Processing,Computer Vision Based Processing and Speed signals is in Minor Detection Region,the automobile is far Extraction.Acoustic Signal Processing first filters the high- away from the mobile phone and the time delay is fixed. frequency signals.Then signals from the top and the bottom This region is of little importance to the speed estimation. microphones will be split into small segments sorted by So we focus on Major Detection Region as shown in the time.The time delays between the segment pairs can be figure 6a.In Major Detection Region,there exist some in- calculated through cross-correlation.We modify time delays valid time delays influenced by environmental noise.There to remove the influence caused by the jitters with inner are two steps to smooth the time delay curve.The first measurement unit(IMU)in the mobile phone.After that step is to replace the invalid delay with some reasonable the time delays form a time delay curve.The time delay values.All the time delays with Ad>Adm should be 36-1233(c)2020 IEEE Personal use is permitted,but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. Authorized licensed use limited to:Nanjing University.Downloaded on December 24,2020 at 09:11:21 UTC from IEEE Xplore.Restrictions apply.1536-1233 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMC.2020.3034354, IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 11 0 5 10 15 20 25 Time(s) -20 -10 0 10 20 Time Delay(sample) Case 1 Case 2 Case 3 34 5 6 87 109 1 2 1211 (a) Time delays of microphones with 12 automo￾biles. 15.5 16 16.5 17 17.5 18 Time(s) -20 -10 0 10 20 Time Delay(sample) Original Calibrated Calibrated by multiple peaks 16.74 (b) Time delays when 7th and 8th automobiles pass by. -500 -250 0 250 500 Time Delay(sample) -2 0 2 Correlation 19 -15 (c) Cross-correlation curve when t = 16.74s. Fig. 13: Processing of acoustic signals of multiple automobiles. in the correlation figures to recover the time delay curve. Then we focus on the correlation between the corresponding segments at time 16.74s. In figure 13a, we have time delay ∆d = 10 around t = 16.4s and time delay ∆d = −5 around t = 16.6s. According to these existing points, we can estimate the range of the time delay when t = 16.74s. Figure 13c shows the cross-correlation of the corresponding segments. Several peaks appear in the figure. We calculate top three peaks in correlation figure. We can see the second highest peak is located on time delay -15 and is suitable for the time delay curve. That means the highest peak at ∆d = 19 is caused by automobile 8 and the second highest peak at ∆d = −15 is caused by automobile 7. At last if no time delay satisfy the range, we will abandon the corresponding segments. Case 3: Only one of the automobiles becomes Major Detection Object when multiple automobiles ao through the Major Detection Region. The situation is shown in figure 11c. In this case the line of sight(LOS) path between the automobiles overlap within the major detection region. That means the delay curve of these automobiles can not be separated from each other. The speed of the automobiles cannot be estimated in this case. In this case, the sound collected by microphones is too complicated to analyze. As a result, we conduct experiments on multiple automobiles and the result shows that if SpeedTalker is not utilized at morning or evening peak, the detection success rate is about 92%. In figure 13a, the curve in red of the 11th and 12th automobiles cannot be distinguished. The two delay curve highly overlap and the two acoustic signals of the automobiles are severely affected by each other. Only the speed of the automobile which can be detected by the cameras can be estimated. 4 SYSTEM DESIGN 4.1 System Overview The system architecture is shown in figure 14. There are three main components in SpeedTalker, i.e., Acoustic Sig￾nal Processing, Computer Vision Based Processing and Speed Extraction. Acoustic Signal Processing first filters the high￾frequency signals. Then signals from the top and the bottom microphones will be split into small segments sorted by time. The time delays between the segment pairs can be calculated through cross-correlation. We modify time delays to remove the influence caused by the jitters with inner measurement unit(IMU) in the mobile phone. After that the time delays form a time delay curve. The time delay curve can be smoothed with Gaussian smoothing. Computer Vision Based Processing first extracts the automobile in the image with yolo[27]. The automobile extracted from the image will be put into a deep learning network to recognize the type of the automobile and know the diameter of its wheel hub in reality. Then we extract the wheel hub through Hough Transform, get the pixel diameter of the wheel hub in the image and estimate the vertical distance between the automobile and the mobile phone. At last, Speed Extraction recovers the trajectory of the automobile and estimates the speed of the automobile. 4.2 Acoustic Signal Processing The acoustic signals we collect usually contain many noises. We can see from figure 2b that the main energy of the sound made by the automobile is distributed in the low￾frequency area. To make the time delay calculated through cross-correlation more accurate, we have let the sound of automobile dominate the acoustic signal. As a result, we use a low-pass filter to remove the high-frequency noises. Dif￾ferent types of automobiles have different noise frequency distribution, but we know that the automobile noise is mainly distributed in the frequencies below 4 kHz. So we use a low-pass filter with the cutoff frequency of 4 kHz. Next, we need to split the acoustic signals into seg￾ments sorted by time. As mentioned before, the size of the segments we design is fs/100. After getting two series of audio segments, we calculate the cross-correlation between the segment pairs to make out the time delay between the corresponding segments. Then we get a series of time delays with timestamp, which can be used to draw the time delay curve. The illustration of the process is shown in figure 15. At last, we need to smooth the time delay curve. In Section 3.2.2, we propose two constraints to filter the time delays. We can divide the whole curve into two parts: Minor Detection Region and Major Detection Region. Minor Detection Region refers to the region where the time delay remains unchanged, with ∆d = ∆dm. The definition of Ma￾jor Detection Region is in Section 3.2.2. When the acoustic signals is in Minor Detection Region, the automobile is far away from the mobile phone and the time delay is fixed. This region is of little importance to the speed estimation. So we focus on Major Detection Region as shown in the figure 6a. In Major Detection Region, there exist some in￾valid time delays influenced by environmental noise. There are two steps to smooth the time delay curve. The first step is to replace the invalid delay with some reasonable values. All the time delays with |∆d| > ∆dm should be Authorized licensed use limited to: Nanjing University. Downloaded on December 24,2020 at 09:11:21 UTC from IEEE Xplore. Restrictions apply