-12 movement in the air will change the air-borne propagation of the sound.Meanwhile,when the finger contacts the surface of the phone,the force applied on the surface will change the vibration pattern of the structure of the phone.which leads Time(seconds) to changes in the structure-borne signal [25].In other words, (a)Magnitude of differential IR estimations when touch and the structure-borne sound is able to distinguish whether the release at 7 cm away from speaker finger is hovering above the surface with a mm-level gap or is pressing on the surface.In VSkin,we mainly use the changes in the structure-borne signal to sense the finger touching, as they provide distinctive information about whether the finger touches the surface or not.However,when force is applied at different locations on the surface,the changes Time(seconds) of the structure-borne sound caused by touching will be (b)Magnitude of differential IR estimations when touch and different in magnitude and phase.Existing schemes only release at 1 cm away from speaker use the magnitude of the structure-borne sound [25],which Figure 10:Touching on different locations has different change rates at different touch positions.They of the given matrix.We assume that the instantaneous cur- rely on the touchscreen to determine the position and the vature remains constant during the time period t-1 ~t and accurate time of the touching to measure the force-level of the phase change of the dynamic component is: touching [25].However,neither the location nor the time =2arcsin (y(t)-y(t1) (7) of the touching is available for VSkin.Therefore,the key 2 challenge in touching sensing for VSkin is to perform joint The path length change for the time period 0~t is: touch detection and touch localization. d0-d0=-2A9x。 (8) Touching events lead to unique patterns in the differential 2π IR estimation.As an example,Figure 10 shows the differen- where di(t)is the path length from the speaker reflected tial IR estimations that are close to the structure-borne path through the finger to the microphone. of the top microphone in Figure 2,when the user touches the back of the phone.The y-axis is the number of samples to the 6.5 From Path Length to Movements structure-borne path,where the structure-borne path(Path The path length change for the reflection air path can 2 in Section 5.3)is at y =0.When force is applied on the be measured on both microphones.Depending on the type surface,the width of the peak corresponding to the structure- of gestures and the placement of the microphones,we can borne path increases.This leads to a small deviation in the use the path length change to derive the actual movement peak position in the path coefficient changes from the orig- distance.For example,for the phone in Figure 2,we can use inal peak of the structure-borne propagation.Figure 10(a) the path length change of the reflection air path on the bottom shows the resulting differential IR estimations when user's microphone to measure the finger movement distance for finger touches/leaves the surface of the mobile device at a the scrolling gesture(up/down movement).This is because position that is 7 cm away from the rear speaker.We observe the length of the reflection path on the bottom microphone that the "hottest"region is not at the original peak of the changes significantly when the finger moves up/down on structure-borne propagation.This is due to the force applied the back of the phone.The actual movement distance can on the surface changes the path of the structure-borne signal. be calculated by multiplying the path length change with a To further explore the change of the structure-borne propa- compensating factor as described in Section 8.For the gesture gation,we ask the user to perform finger tapping on eleven of swiping left/right,we can use path length changes of two different positions on the back of the device and measure microphones to determine the swiping direction,as swiping the position of peaks in the path coefficient changes.Figure left and right will introduce the same path length change 11 shows the relationship between the touching position pattern on the bottom microphone but different path length and the resulting peak position in coefficient changes,where change directions on the top microphone. the peak position is measured by the number of samples to the original structure-borne path.We observe that the 7 TOUCH MEASUREMENT larger the distance between the touching position and the 7.1 Touch Signal Pattern speaker,the larger the delay in coefficient changes to the Touching the surface with fingers will change both the original structure-borne path (darker color means a larger air-borne propagation and structure-borne propagation of delay).Thus,we utilize the magnitude and delay of differen- the sound.When performing the tapping action,the finger tial IR estimations to detect and localize touch events.Note0123 Time (seconds) -12 0 12 24 Delay samples (a) Magnitude of differential IR estimations when touch and release at 7 cm away from speaker 0123 Time (seconds) -12 0 12 24 Delay samples (b) Magnitude of differential IR estimations when touch and release at 1 cm away from speaker Figure 10: Touching on different locations of the given matrix. We assume that the instantaneous cur￾vature remains constant during the time period t − 1 ∼ t and the phase change of the dynamic component is: ∆θ t t−1 = 2 arcsin k(t)  y(t) − y(t − 1)   2 . (7) The path length change for the time period 0 ∼ t is: di (t) − di (0) = − Pt i=1 ∆θ i i−1 2π × λc , (8) where di (t) is the path length from the speaker reflected through the finger to the microphone. 6.5 From Path Length to Movements The path length change for the reflection air path can be measured on both microphones. Depending on the type of gestures and the placement of the microphones, we can use the path length change to derive the actual movement distance. For example, for the phone in Figure 2, we can use the path length change of the reflection air path on the bottom microphone to measure the finger movement distance for the scrolling gesture (up/down movement). This is because the length of the reflection path on the bottom microphone changes significantly when the finger moves up/down on the back of the phone. The actual movement distance can be calculated by multiplying the path length change with a compensating factor as described in Section 8. For the gesture of swiping left/right, we can use path length changes of two microphones to determine the swiping direction, as swiping left and right will introduce the same path length change pattern on the bottom microphone but different path length change directions on the top microphone. 7 TOUCH MEASUREMENT 7.1 Touch Signal Pattern Touching the surface with fingers will change both the air-borne propagation and structure-borne propagation of the sound. When performing the tapping action, the finger movement in the air will change the air-borne propagation of the sound. Meanwhile, when the finger contacts the surface of the phone, the force applied on the surface will change the vibration pattern of the structure of the phone, which leads to changes in the structure-borne signal [25]. In other words, the structure-borne sound is able to distinguish whether the finger is hovering above the surface with a mm-level gap or is pressing on the surface. In VSkin, we mainly use the changes in the structure-borne signal to sense the finger touching, as they provide distinctive information about whether the finger touches the surface or not. However, when force is applied at different locations on the surface, the changes of the structure-borne sound caused by touching will be different in magnitude and phase. Existing schemes only use the magnitude of the structure-borne sound [25], which has different change rates at different touch positions. They rely on the touchscreen to determine the position and the accurate time of the touching to measure the force-level of touching [25]. However, neither the location nor the time of the touching is available for VSkin. Therefore, the key challenge in touching sensing for VSkin is to perform joint touch detection and touch localization. Touching events lead to unique patterns in the differential IR estimation. As an example, Figure 10 shows the differen￾tial IR estimations that are close to the structure-borne path of the top microphone in Figure 2, when the user touches the back of the phone. The y-axis is the number of samples to the structure-borne path, where the structure-borne path (Path 2 in Section 5.3) is at y = 0. When force is applied on the surface, the width of the peak corresponding to the structure￾borne path increases. This leads to a small deviation in the peak position in the path coefficient changes from the orig￾inal peak of the structure-borne propagation. Figure 10(a) shows the resulting differential IR estimations when user’s finger touches/leaves the surface of the mobile device at a position that is 7 cm away from the rear speaker. We observe that the “hottest” region is not at the original peak of the structure-borne propagation. This is due to the force applied on the surface changes the path of the structure-borne signal. To further explore the change of the structure-borne propa￾gation, we ask the user to perform finger tapping on eleven different positions on the back of the device and measure the position of peaks in the path coefficient changes. Figure 11 shows the relationship between the touching position and the resulting peak position in coefficient changes, where the peak position is measured by the number of samples to the original structure-borne path. We observe that the larger the distance between the touching position and the speaker, the larger the delay in coefficient changes to the original structure-borne path (darker color means a larger delay). Thus, we utilize the magnitude and delay of differen￾tial IR estimations to detect and localize touch events. Note