GUO ET AL.:MESH-GUIDED OPTIMIZED RETEXTURING FOR IMAGE AND VIDEO 427 texture replacements are conducted [4],[5].Second,the Since tracking objects throughout video sequence is one generated results are propagated to the rest frames.These important step of our approach,it is briefly reviewed. methods either need cumbersome interaction to locate the region of interest(ROD)frame by frame [4]or utilize special 2.1 Image Texture Replacement color-coded pattern as input [5]. The pioneering work on texture replacement dealt with In this paper,we propose one novel approach for extracting lighting map from given image [3].Based on optimized retexturing on image and video.For image certain lighting distribution models,Tsin et al.introduced texture replacement,we formulate texture distortion as one one Bayesian framework for near-regular texture,which stretch-based parameterization.The ROI is represented as a relies on the color observation at each pixel [3].Oh et al. feature-mesh coupled with normal field.The corresponding assumed that large-scale luminance variations are due to mesh of the feature-mesh is computed in texture space geometry and lighting [6]and presented an algorithm for during parameterization.For simulating the effect of texture decoupling texture luminance from image by applying an deformation at fine scale,one Poisson-based refinement improved bilateral filter.Currently,accurate recovery of process is developed.Based on our image-retexturing lighting from natural image is still a challenging problem. scheme,we design one key-frame-based video retexturing Assuming that object appearance satisfies the Lamber- tian reflectance model,Textureshop [1]recovered normal approach similar to RotoTexture [4].Once replacing field of specified region using a simplified shape-from- textures of the specified regions of key frames,these shading algorithm [7].One propagation rule of adjacent generated effects are iteratively transferred to the rest texture coordinates is deduced to guide a normal-related frames.For achieving temporal coherence,mesh points of synthesis process.The limitation of employing texture key frame serve as features that are tracked and further synthesis is that the synthesis process must be re-executed optimized using motion analysis as well over the whole when a new texture is applied.The work developed by image sequence through temporal smoothing.Graphcut Zelinka et al.[8]is an improvement over Textureshop.It segmentation is adopted for handling object occlusions by reduces user interaction of object specification by employ- extracting new appearing parts in each frame. ing efficient object cutout algorithm [9].In addition,jump Our optimized retexturing approach has the following map-based synthesis [10]is adopted to speed up the new features: computation process.Instead of texture synthesis,our method belongs to the texture mapping approach.Texture ● A two-step mesh guided process for image texture replacement can be efficiently carried out after the mesh replacement.Coupled with recovered normal field, parameterization is completed. visually pleasing deformation effect of the replaced The method presented in [4]warps an entire texture onto texture is produced by performing stretch-based photographed surface.It minimizes one energy function of mesh parameterization.Furthermore,a Poisson- a spring network with known evenly distributed rectilinear based refinement process is used to improve the grid in texture space.In most cases,the specified region of effect and enhance the efficiency. the image is a usually irregular grid.Hence,it is difficult for Creation of special retexturing effects.Based on mesh this approach to accurately control the mapping position of parameterization,we can easily generate a replace- the replaced texture. ment effect with progressively variant texton scales. For extracting deformation fields of textures in natural In addition,texture discontinuities can be realisti- images,Liu et al.introduced one user-assisted adjustment cally simulated in self-occlusion regions,which are scheme on the regular lattice of real texture [2].A bijective usually difficult to produce for most previous mapping between the regular lattice and its deformed approaches. shape on the surface image is obtained.Any new texture An optimized framework of video retexturing.We extend can thus be replaced onto the source image by exerting the and generalize our image retexturing approach to video.Rather than presegmenting the ROI through- corresponding mapping.Since this method often requires elaborate user interaction,it is more suitable for regular/ out the whole image sequence,our approach only near-regular textures. needs to select a few of key frames.The generated results are optimally propagated to the rest frames. Besides image texture replacement,recent research Texture drifting and visibility shifting are also demonstrated that material properties of objects can be changed in image space [11].Exploiting the fact that human tackled effectively. vision is surprisingly tolerant of certain physical inaccura- The rest of the paper is organized as follows:The related cies,Khan et al.reconstructed depth map of the concerned work is described in Section 2.Our optimized retexturing object with other environment parameters [11]and realized approach for image is presented in Section 3.In Section 4, compelling material editing effects using complex relight- the image retexturing approach is extended and generalized ing techniques to video.The experimental results are presented in Section 5. Finally,we draw conclusions and point out the future work. 2.2 Video Texture Replacement Rototexture [4]generalized the method of Textureshop [1] 2 RELATED WORK to video.It provides two means of texturing a raw video sequence,namely,texture mapping and texture synthesis. This paper is made possible by many inspirations from The texture mapping method uses one nonlinear optimiza- previous work on image and video texture replacement.tion of a spring model to control the behavior of texturetexture replacements are conducted [4], [5]. Second, the generated results are propagated to the rest frames. These methods either need cumbersome interaction to locate the region of interest (ROI) frame by frame [4] or utilize special color-coded pattern as input [5]. In this paper, we propose one novel approach for optimized retexturing on image and video. For image texture replacement, we formulate texture distortion as one stretch-based parameterization. The ROI is represented as a feature-mesh coupled with normal field. The corresponding mesh of the feature-mesh is computed in texture space during parameterization. For simulating the effect of texture deformation at fine scale, one Poisson-based refinement process is developed. Based on our image-retexturing scheme, we design one key-frame-based video retexturing approach similar to RotoTexture [4]. Once replacing textures of the specified regions of key frames, these generated effects are iteratively transferred to the rest frames. For achieving temporal coherence, mesh points of key frame serve as features that are tracked and further optimized using motion analysis as well over the whole image sequence through temporal smoothing. Graphcut segmentation is adopted for handling object occlusions by extracting new appearing parts in each frame. Our optimized retexturing approach has the following new features: . A two-step mesh guided process for image texture replacement. Coupled with recovered normal field, visually pleasing deformation effect of the replaced texture is produced by performing stretch-based mesh parameterization. Furthermore, a Poisson￾based refinement process is used to improve the effect and enhance the efficiency. . Creation of special retexturing effects. Based on mesh parameterization, we can easily generate a replace￾ment effect with progressively variant texton scales. In addition, texture discontinuities can be realisti￾cally simulated in self-occlusion regions, which are usually difficult to produce for most previous approaches. . An optimized framework of video retexturing. We extend and generalize our image retexturing approach to video. Rather than presegmenting the ROI through￾out the whole image sequence, our approach only needs to select a few of key frames. The generated results are optimally propagated to the rest frames. Texture drifting and visibility shifting are also tackled effectively. The rest of the paper is organized as follows: The related work is described in Section 2. Our optimized retexturing approach for image is presented in Section 3. In Section 4, the image retexturing approach is extended and generalized to video. The experimental results are presented in Section 5. Finally, we draw conclusions and point out the future work. 2 RELATED WORK This paper is made possible by many inspirations from previous work on image and video texture replacement. Since tracking objects throughout video sequence is one important step of our approach, it is briefly reviewed. 2.1 Image Texture Replacement The pioneering work on texture replacement dealt with extracting lighting map from given image [3]. Based on certain lighting distribution models, Tsin et al. introduced one Bayesian framework for near-regular texture, which relies on the color observation at each pixel [3]. Oh et al. assumed that large-scale luminance variations are due to geometry and lighting [6] and presented an algorithm for decoupling texture luminance from image by applying an improved bilateral filter. Currently, accurate recovery of lighting from natural image is still a challenging problem. Assuming that object appearance satisfies the Lamber￾tian reflectance model, Textureshop [1] recovered normal field of specified region using a simplified shape-from￾shading algorithm [7]. One propagation rule of adjacent texture coordinates is deduced to guide a normal-related synthesis process. The limitation of employing texture synthesis is that the synthesis process must be re-executed when a new texture is applied. The work developed by Zelinka et al. [8] is an improvement over Textureshop. It reduces user interaction of object specification by employ￾ing efficient object cutout algorithm [9]. In addition, jump map-based synthesis [10] is adopted to speed up the computation process. Instead of texture synthesis, our method belongs to the texture mapping approach. Texture replacement can be efficiently carried out after the mesh parameterization is completed. The method presented in [4] warps an entire texture onto photographed surface. It minimizes one energy function of a spring network with known evenly distributed rectilinear grid in texture space. In most cases, the specified region of the image is a usually irregular grid. Hence, it is difficult for this approach to accurately control the mapping position of the replaced texture. For extracting deformation fields of textures in natural images, Liu et al. introduced one user-assisted adjustment scheme on the regular lattice of real texture [2]. A bijective mapping between the regular lattice and its deformed shape on the surface image is obtained. Any new texture can thus be replaced onto the source image by exerting the corresponding mapping. Since this method often requires elaborate user interaction, it is more suitable for regular/ near-regular textures. Besides image texture replacement, recent research demonstrated that material properties of objects can be changed in image space [11]. Exploiting the fact that human vision is surprisingly tolerant of certain physical inaccura￾cies, Khan et al. reconstructed depth map of the concerned object with other environment parameters [11] and realized compelling material editing effects using complex relight￾ing techniques. 2.2 Video Texture Replacement Rototexture [4] generalized the method of Textureshop [1] to video. It provides two means of texturing a raw video sequence, namely, texture mapping and texture synthesis. The texture mapping method uses one nonlinear optimiza￾tion of a spring model to control the behavior of texture GUO ET AL.: MESH-GUIDED OPTIMIZED RETEXTURING FOR IMAGE AND VIDEO 427