This article has been accepted for publication in a future issue of this journal,but has not been fully edited.Content may change prior to final publication.Citation information:DOI 10.1109/TMC.2020.3034354.IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING,VOL.XX,NO.XX,2020 3 different noises with two built-in microphones of mobile phones are at the sidewalk and the positions and gestures phones.On the other hand,there might be many kinds are unknown.So novel approaches utilizing mobile phones of noises in the environment,especially for the sound of to calculate the speed of the automobiles are needed.To other automobiles on the road.It is hard to remove the get the relative position information between automobiles environment noises,since the frequencies of other auto- and mobile phones,we need to use cameras inside the mobiles mainly lie in very close frequency band with the mobile phones,which is analogous to knowing the posi- target automobile.To address this challenge,we consider tion and gestures of the cameras in traditional CV based the acoustic signals at full frequency as a whole.We utilize approaches.Apart from distance calculation,SpeedTalker the cross-correlation of the acoustic signals from the top utilizes acoustic signals to estimate the candidate trajectory and bottom microphones to estimate the time difference of of the automobiles.There are two advantages of acoustic arrivals(TDOA).As the automobile is continuously moving, signals over the visual signals.Firstly,the detection region we can obtain a series of time delays through TDOA at of acoustic signals is broader than that of visual signals. different time.The candidate trajectories of the automobile Common cameras inside the microphones usually have nar- can be estimated as a set of hyperbolas according to the row field of view(FOV).For example,the wide-angle camera curve of the time delay.Thus the automobile speed can be of Samsung Galaxy Note 8 has 77 field of view.If we further estimated. utilize the microphones of Samsung Galaxy Note 8 to detect The third challenge is to estimate the speed of multiple automobiles,the detection field of view is around 160 automobiles.We can not separate the sound of multiple according to the hyperbola model we propose.Secondly, automobiles.Therefore,when multiple automobiles pass compute complexity of acoustic signals processing is much through the mobile phone,it is challenging to estimate the lower than that of visual signals.If visual signals are utilized speed.To address the challenge,we utilize the multiple to complete the same work,each frame of the videos should peaks in the cross-correlation figures between the top and be processed.The compute complexity of the processing is bottom microphones.Then we may recover the delay curve unacceptable. of each automobile and calculate the speed of the automo- Automobile detection via mobile phones:Automobile biles. detection is an important research area since undetected automobiles are likely to endanger human life.Mobile 1.5 Contributions phones can be utilized to inform the users of the approach- This paper makes four contributions:First,this is the first ing automobiles.There are three approaches to sense the work that estimates the automobile speed via mobile phones automobiles with mobile phones.The first approach is to through passive sensing of acoustic and image signals.We install applications both on the automobiles and the mobile propose an integrated solution to effectively estimate the au- phones.Oki Electric Industry Co.Ltd.develops a mobile tomobile's speed based on commercial off-the-shelf(COTS) phone that notifies the users of the presence of the auto- devices,and provide a platform for every pedestrian to mobiles using DSRC[9].Car-2-X utilizes ad-hoc and cellular help report the speeding event of automobiles.Second,we networks to inform the pedestrians of the automobile with use the time difference of arrivals (TDOA)model based on the same method[10].The second approach is to sense acoustic signals to figure out the candidate trajectories of the moving automobiles via images.Sivaraman proposed a automobile,and use the pin-hole model based on image general active-learning framework for on-road automobiles frames to figure out the vertical distance,thus to estimate recognition and tracking based on videos[11].Wang pro- the unique trajectory.Combined with the timestamp of the posed WalkSafe,a mobile phone application based on the trajectory,the automobile speed can be estimated.Third,we back camera to sense the automobiles[121.The drawback of implemented a system prototype for SpeedTalker and esti- these work is that image processing needs huge calculating mated the automobile speed with high accuracy.The system resources.And the camera of the mobile phone is needed to works in the outdoor environment and effectively mitigates face the road,which makes the detection inconvenient.The the ambient environmental interference.Experiment results third approach is to utilize acoustic signals to sense the auto- show that SpeedTalker can achieve an average estimation mobiles.Tsuzuki proposed an automobile sound detection error of 6.1%in the scenario of single automobile.In the system for a mobile phone[13].Takagi introduced a hybrid scenario of multiple automobiles,SpeedTalker can achieve and electric vehicles detection system[14],which focused an average estimation error of 9.8%. on switching noise of the electric motor.So they failed to detect automobiles other than these types.Li proposed Auto++,a system that detects approaching automobiles for 2 RELATED WORK smart phone users to detect all kinds of automobiles via Automobile detection via visual signals:Traditional ap- overall acoustic signals[15].However,all these works can proaches utilize cameras to calculate the speed of the au- only inform the user of the approach of the automobiles tomobiles.Kumar7]and Czajewski8]use computer vision and can not estimate the speed of the automobile. based technologies to detect automobiles.The cameras are Sensing via acoustic signals with mobile phones:Sens- deployed in fixed positions and gestures above the street. ing with daily equipment is a popular issue.Sound waves As a result,the detection region is known and fixed.That can easily be transmitted and received by daily equipment, means the moving distance of the automobiles can easily such as mobile phones and smart watches.Much work be acquired.Then the speed of the automobiles can be based on sound wave has been published.AAMouse mea- calculated.However,the scenarios of SpeedTalker is differ- sures the Doppler Shift of the sound waves transmitted by ent from that of traditional visual approaches.The mobile a mobile phone to track the phone itself with an accuracy 1536-1233 (c)2020 IEEE.Personal use is permitted,but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. Authorized licensed use limited to:Nanjing University.Downloaded on December 24,2020 at 09:11:21 UTC from IEEE Xplore.Restrictions apply.1536-1233 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information. This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TMC.2020.3034354, IEEE Transactions on Mobile Computing IEEE TRANSACTIONS ON MOBILE COMPUTING, VOL. XX, NO. XX, 2020 3 different noises with two built-in microphones of mobile phones. On the other hand, there might be many kinds of noises in the environment, especially for the sound of other automobiles on the road. It is hard to remove the environment noises, since the frequencies of other auto￾mobiles mainly lie in very close frequency band with the target automobile. To address this challenge, we consider the acoustic signals at full frequency as a whole. We utilize the cross-correlation of the acoustic signals from the top and bottom microphones to estimate the time difference of arrivals (TDOA). As the automobile is continuously moving, we can obtain a series of time delays through TDOA at different time. The candidate trajectories of the automobile can be estimated as a set of hyperbolas according to the curve of the time delay. Thus the automobile speed can be further estimated. The third challenge is to estimate the speed of multiple automobiles. We can not separate the sound of multiple automobiles. Therefore, when multiple automobiles pass through the mobile phone, it is challenging to estimate the speed. To address the challenge, we utilize the multiple peaks in the cross-correlation figures between the top and bottom microphones. Then we may recover the delay curve of each automobile and calculate the speed of the automo￾biles. 1.5 Contributions This paper makes four contributions: First, this is the first work that estimates the automobile speed via mobile phones through passive sensing of acoustic and image signals. We propose an integrated solution to effectively estimate the au￾tomobile’s speed based on commercial off-the-shelf(COTS) devices, and provide a platform for every pedestrian to help report the speeding event of automobiles. Second, we use the time difference of arrivals (TDOA) model based on acoustic signals to figure out the candidate trajectories of automobile, and use the pin-hole model based on image frames to figure out the vertical distance, thus to estimate the unique trajectory. Combined with the timestamp of the trajectory, the automobile speed can be estimated. Third, we implemented a system prototype for SpeedTalker and esti￾mated the automobile speed with high accuracy. The system works in the outdoor environment and effectively mitigates the ambient environmental interference. Experiment results show that SpeedTalker can achieve an average estimation error of 6.1% in the scenario of single automobile. In the scenario of multiple automobiles, SpeedTalker can achieve an average estimation error of 9.8%. 2 RELATED WORK Automobile detection via visual signals: Traditional ap￾proaches utilize cameras to calculate the speed of the au￾tomobiles. Kumar[7] and Czajewski[8] use computer vision based technologies to detect automobiles. The cameras are deployed in fixed positions and gestures above the street. As a result, the detection region is known and fixed. That means the moving distance of the automobiles can easily be acquired. Then the speed of the automobiles can be calculated. However, the scenarios of SpeedTalker is differ￾ent from that of traditional visual approaches. The mobile phones are at the sidewalk and the positions and gestures are unknown. So novel approaches utilizing mobile phones to calculate the speed of the automobiles are needed. To get the relative position information between automobiles and mobile phones, we need to use cameras inside the mobile phones, which is analogous to knowing the posi￾tion and gestures of the cameras in traditional CV based approaches. Apart from distance calculation, SpeedTalker utilizes acoustic signals to estimate the candidate trajectory of the automobiles. There are two advantages of acoustic signals over the visual signals. Firstly, the detection region of acoustic signals is broader than that of visual signals. Common cameras inside the microphones usually have nar￾row field of view(FOV). For example, the wide-angle camera of Samsung Galaxy Note 8 has 77◦ field of view. If we utilize the microphones of Samsung Galaxy Note 8 to detect automobiles, the detection field of view is around 160◦ according to the hyperbola model we propose. Secondly, compute complexity of acoustic signals processing is much lower than that of visual signals. If visual signals are utilized to complete the same work, each frame of the videos should be processed. The compute complexity of the processing is unacceptable. Automobile detection via mobile phones: Automobile detection is an important research area since undetected automobiles are likely to endanger human life. Mobile phones can be utilized to inform the users of the approach￾ing automobiles. There are three approaches to sense the automobiles with mobile phones. The first approach is to install applications both on the automobiles and the mobile phones. Oki Electric Industry Co. Ltd. develops a mobile phone that notifies the users of the presence of the auto￾mobiles using DSRC[9]. Car-2-X utilizes ad-hoc and cellular networks to inform the pedestrians of the automobile with the same method[10]. The second approach is to sense the moving automobiles via images. Sivaraman proposed a general active-learning framework for on-road automobiles recognition and tracking based on videos[11]. Wang pro￾posed WalkSafe, a mobile phone application based on the back camera to sense the automobiles[12]. The drawback of these work is that image processing needs huge calculating resources. And the camera of the mobile phone is needed to face the road, which makes the detection inconvenient. The third approach is to utilize acoustic signals to sense the auto￾mobiles. Tsuzuki proposed an automobile sound detection system for a mobile phone[13]. Takagi introduced a hybrid and electric vehicles detection system[14], which focused on switching noise of the electric motor. So they failed to detect automobiles other than these types. Li proposed Auto++, a system that detects approaching automobiles for smart phone users to detect all kinds of automobiles via overall acoustic signals[15]. However, all these works can only inform the user of the approach of the automobiles and can not estimate the speed of the automobile. Sensing via acoustic signals with mobile phones: Sens￾ing with daily equipment is a popular issue. Sound waves can easily be transmitted and received by daily equipment, such as mobile phones and smart watches. Much work based on sound wave has been published. AAMouse mea￾sures the Doppler Shift of the sound waves transmitted by a mobile phone to track the phone itself with an accuracy Authorized licensed use limited to: Nanjing University. Downloaded on December 24,2020 at 09:11:21 UTC from IEEE Xplore. Restrictions apply