Image completion based on views of large displacement 837 Gaps Warped PSR】 Warped PSRI Warped PSRI Homographyl PSR PSRI Separate Transformation PSR2 PSR2 Homography2 2 Warped PSR2 Warped PSR2 PSR2 Warped PSR2 Overlappings The target region 6 The practical strategy Fig.3.Transformation of the visible information on the LDV Fig.4a,b.Our image repairing algorithm.The target image con- image.The candidate PSRs on the LDV image are independently sists of two PSRs with their common boundary Lsh.The warped warped onto their counterparts on the target image,hence several PSRs fall onto the repairing window 22,which encloses the missing overlappings and gaps appear among the warped candidate PSRs region s2 on I.a Circled by the blue lines,22 contains three parts. 1.e.,the missing pixels under the overlappings 222,the gaps 2 and the remainder 22.b Our algorithm divides 2 into three parts and Graph cut based image stitching stitches the warped repairs these with different schemes.The missing pixels in 2a out- candidate PSRs onto 2.. side the region of band are first recovered by graph cut based image stitching.Then those in 2 are restored by texture synthe- -Texture synthesis based image inpainting samples the sis based image inpainting.Finally,those in sc are completed by warped PSRs to synthesize 2 in terms of a defined image fusion based hole filling repairing priority function. -Image fusion based hole filling fuses the nearby known pixels to fill the gaps 2c under the control of a defined date matches for p are confined to the surrounding pixels fusion priority function. of the corresponding 3 x 3 patch in the warped candidate PSRs.The whole process is terminated when a certain We now discuss the details of our repairing algorithm. condition is satisfied. To ensure a faithful restoration of the potential feature 3.2.1 Graph cut based image stitching lines along the common boundary Lsb,it is important to select a good seed pixel to start the texture synthesis pro- In order to stitch the warped candidate PSRs onto a cess.An appropriate repairing priority function needs to seamlessly,we use a graph cut algorithm [4]to discover be defined.For a boundary pixel p in the missing region, the optimal seam lines in the overlapping regions of the we define its repairing priority as Pr(p)=Ci(p)*Si(p), warped PSRs and the known area on the target image where within 22.The graph cut algorithm works by expressing the problem as finding the min-cut in a weighted graph and minimizing the color gradient differences across the C(p)=〉w(pi)/8 seaminess.This has already been used in texture synthe- i=1 sis [16]and image stitching [25],and is well suited for this purpose. is the confidence term that represents the reliable informa- After finding the optimal seaminess,the warped PSRs tion contained in p's 8-neighborhood,thereinto w(pi)is are stitched onto the target image along the seaminess to the reliability weight of p's 8-neighborhood pixel pi.In fill the remaining pixels in 22a. implementation,we set a threshold t=3/8 for Ci(p)to qualify the boundary pixels with enough known neighbors 3.2.2 Texture synthesis based image inpainting and control the repairing process from the boundary pixels to the interior of 2'progressively. The remaining missing pixels in o are filled one by one with texture synthesis based image inpainting.It begins S(p)=(/L)*(7Ih·np/a) with a pixel on the boundary of 22',i.e.,022',and iter- atively selects the next pixel with the highest priority to is the structure term,in which is the number of PSRs re- proceed.For each pixel p to be filled,we define a 3x 3 ferred to by pixels in p's 8-neighborhood.L is the number patch centered at p.The existent pixels in the patch are of warped PSRs projecting into the repairing window.l/L used as the constraints for finding the best match m ac- can endow the pixels near Lsb with high repairing priority. cording to the SSD metric (the sum of squared differences VIp represents the color gradient at p,L denotes the orth- between the colors of their surrounding pixels).The candi- ogonal operator,np is the unit normal of p on as2'and aImage completion based on views of large displacement 837 Fig. 3. Transformation of the visible information on the LDV image. The candidate PSRs on the LDV image are independently warped onto their counterparts on the target image, hence several overlappings and gaps appear among the warped candidate PSRs – Graph cut based image stitching stitches the warped candidate PSRs onto Ωa. – Texture synthesis based image inpainting samples the warped PSRs to synthesize Ωb in terms of a defined repairing priority function. – Image fusion based hole filling fuses the nearby known pixels to fill the gaps Ωc under the control of a defined fusion priority function. We now discuss the details of our repairing algorithm. 3.2.1 Graph cut based image stitching In order to stitch the warped candidate PSRs onto Ωa seamlessly, we use a graph cut algorithm [4] to discover the optimal seam lines in the overlapping regions of the warped PSRs and the known area on the target image within Ωˆ . The graph cut algorithm works by expressing the problem as finding the min-cut in a weighted graph and minimizing the color gradient differences across the seaminess. This has already been used in texture synthe￾sis [16] and image stitching [25], and is well suited for this purpose. After finding the optimal seaminess, the warped PSRs are stitched onto the target image along the seaminess to fill the remaining pixels in Ωa. 3.2.2 Texture synthesis based image inpainting The remaining missing pixels in Ωb are filled one by one with texture synthesis based image inpainting. It begins with a pixel on the boundary of Ω
, i.e., ∂Ω
, and iter￾atively selects the next pixel with the highest priority to proceed. For each pixel p to be filled, we define a 3×3 patch centered at p. The existent pixels in the patch are used as the constraints for finding the best match m ac￾cording to the SSD metric (the sum of squared differences between the colors of their surrounding pixels). The candi￾Fig. 4a,b. Our image repairing algorithm. The target image I con￾sists of two PSRs with their common boundary Lsb. The warped PSRs fall onto the repairing window Ωˆ , which encloses the missing region Ω on I. a Circled by the blue lines, Ω contains three parts, i.e., the missing pixels under the overlappings Ωo b , the gaps Ωo c and the remainder Ωo a . b Our algorithm divides Ω into three parts and repairs these with different schemes. The missing pixels in Ωa out￾side the region of band Ω
are first recovered by graph cut based image stitching. Then those in Ωb are restored by texture synthe￾sis based image inpainting. Finally, those in Ωc are completed by image fusion based hole filling date matches for p are confined to the surrounding pixels of the corresponding 3×3 patch in the warped candidate PSRs. The whole process is terminated when a certain condition is satisfied. To ensure a faithful restoration of the potential feature lines along the common boundary Lsb, it is important to select a good seed pixel to start the texture synthesis pro￾cess. An appropriate repairing priority function needs to be defined. For a boundary pixel p in the missing region, we define its repairing priority as PI(p) = CI(p) ∗ SI(p), where CI(p) = 8 i=1 w(pi)/8 is the confidence term that represents the reliable informa￾tion contained in p’s 8-neighborhood, thereinto w(pi) is the reliability weight of p’s 8-neighborhood pixel pi. In implementation, we set a threshold τ = 3/8 for CI(p) to qualify the boundary pixels with enough known neighbors and control the repairing process from the boundary pixels to the interior of Ω
progressively. SI(p) = (l/L) ∗ (∇I⊥ p · np/α) is the structure term, in which l is the number of PSRs re￾ferred to by pixels in p’s 8-neighborhood. L is the number of warped PSRs projecting into the repairing window. l/L can endow the pixels near Lsb with high repairing priority. ∇Ip represents the color gradient at p, ⊥ denotes the orth￾ogonal operator, np is the unit normal of p on ∂Ω
and α