COMPUTER ANIMATION AND VIRTUAL WORLDS Comp.Anim.Virtual Worlds 2005:16:451-461 Published online in Wiley InterScience (www.interscience.wiley.com).DOl:10.1002/cav.82 Image,Colour and Illumination in Animation Image and video retexturing By Yanwen Guo*,Jin Wang,Xiang Zeng,Zhongyi Xie, Hanqiu Sun and Qunsheng Peng We propose a novel imagelvideo retexturing approach that preserves the original shading effects without knowing the underlying surface and lighting conditions.For static images, we first introduce the Poisson equation-based algorithm to simulate the texture distortion on the projected interest region of the underlying surface,while preserving the shading effect of the original image.We further work on videos by retexturing the key frame as static image and then propagating the results onto the other frames.In video retexturing,we have introduced the mesh based optimization for object tracking to avoid texture drifting,and the graph cut algorithm to effectively deal with visibility shift between frames.The graph cut algorithm is applied on a trimap along the boundary of the object to extract the textured part inside the trimap.The proposed approach is developed in imagelvideo retexturing at nearly interactive rate,and our experimental results have showed the satisfactory performance of our approach.Copyright C 2005 Jol Wiley Sons,Ltd. KEY WORDS:image/video retexturing;Poisson equation;graph cut Introduction with the same motivations.In this paper,we propose a novel image/video retexturing approach while preser- Retexturing is the process of replacing existing textures ving the original shading effects. in the concerned region of images/videos with new Manipulating textures in real images has been fasci- ones while preserving the original shading effects.It nating people for a long time,and the understanding of has wide applications in special effects in TV and film texture evolves meanwhile.Early works,model texture producing,art and industrial design,distance learning, as a statistical attributel of a surface and decompose digital entertainment,and E-commence.To achieve real-world texture into a texture part and a lighting realistic retexturing effects,two basic problems must part.2 Recent studies categorize real-world textures be solved.One is to make the new texture adequately into regular and irregular types3 and decompose the wrapped and shaded so that it is consistent with the texture into geometry,lighting,and color components. unknown shape of the underlying surface as well as the Nevertheless,recovering the geometry,lighting compo- unknown lighting condition encoded by the original nents of a texture in a single image is very difficult. image.The other is how to prevent the new texture Because real-world images are usually taken under very drifting on the interested region between adjacent complex environment,physically based techniques like frames in video.The related research is mainly pro- shape-from-shading (SFS)are sometimes complex,un- posed for the problems individually,but not the pro- stable,and inaccurate. blems simultaneously for both image/video retexturing Rather than recovering the geometry and lighting information,our retexturing approach of images aims at producing an illusion such that the replaced texture inherits the geometry and lighting information implied *Correspondence to:Yanwen Guo,State Key Lab of CAD&CG, in the input image.By solving the Poisson equations,a Department of Mathematics,Zhejiang University,Hangzhou, China.E-mail:ywguo@cad.zju.edu.cn non-linear mapping between the new texture and the concerned region on the image is derived reflecting the Contract/grant sponsor:973 Program of China;contract/grant original shape of the underlying surface.Lighting effect number:2002CB312101. Contract/grant sponsor:NSFC;contract/grant numbers: on the original image is retained by adopting the YCbCr 60033010:60403038. color space to represent the previous texture value at 带事带年年海带带年事泰市年年带垂年带甲海垂事布而垂专年带带事香带带年带布车香甲海垂事布而年 Copyright C 2005 John Wiley Sons,Ltd
COMPUTER ANIMATION AND VIRTUAL WORLDS Comp. Anim. Virtual Worlds 2005; 16: 451–461 Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/cav.82 ****************************************************************************************************** Image, Colour and Illumination in Animation Image and video retexturing By Yanwen Guo*, Jin Wang, Xiang Zeng, Zhongyi Xie, Hanqiu Sun and Qunsheng Peng ****************************************************************************************************** We propose a novel image/video retexturing approach that preserves the original shading effects without knowing the underlying surface and lighting conditions. For static images, we first introduce the Poisson equation-based algorithm to simulate the texture distortion on the projected interest region of the underlying surface, while preserving the shading effect of the original image. We further work on videos by retexturing the key frame as static image and then propagating the results onto the other frames. In video retexturing, we have introduced the mesh based optimization for object tracking to avoid texture drifting, and the graph cut algorithm to effectively deal with visibility shift between frames. The graph cut algorithm is applied on a trimap along the boundary of the object to extract the textured part inside the trimap. The proposed approach is developed in image/video retexturing at nearly interactive rate, and our experimental results have showed the satisfactory performance of our approach. Copyright # 2005 John Wiley & Sons, Ltd. KEY WORDS: image/video retexturing; Poisson equation; graph cut Introduction Retexturing is the process of replacing existing textures in the concerned region of images/videos with new ones while preserving the original shading effects. It has wide applications in special effects in TV and film producing, art and industrial design, distance learning, digital entertainment, and E-commence. To achieve realistic retexturing effects, two basic problems must be solved. One is to make the new texture adequately wrapped and shaded so that it is consistent with the unknown shape of the underlying surface as well as the unknown lighting condition encoded by the original image. The other is how to prevent the new texture drifting on the interested region between adjacent frames in video. The related research is mainly proposed for the problems individually, but not the problems simultaneously for both image/video retexturing with the same motivations. In this paper, we propose a novel image/video retexturing approach while preserving the original shading effects. Manipulating textures in real images has been fascinating people for a long time, and the understanding of texture evolves meanwhile. Early works, model texture as a statistical attribute1 of a surface and decompose real-world texture into a texture part and a lighting part.2 Recent studies categorize real-world textures into regular and irregular types3,4 and decompose the texture into geometry, lighting, and color components. Nevertheless, recovering the geometry, lighting components of a texture in a single image is very difficult. Because real-world images are usually taken under very complex environment, physically based techniques like shape-from-shading (SFS) are sometimes complex, unstable, and inaccurate. Rather than recovering the geometry and lighting information, our retexturing approach of images aims at producing an illusion such that the replaced texture inherits the geometry and lighting information implied in the input image. By solving the Poisson equations, a non-linear mapping between the new texture and the concerned region on the image is derived reflecting the original shape of the underlying surface. Lighting effect on the original image is retained by adopting the YCbCr color space to represent the previous texture value at ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. *Correspondence to: Yanwen Guo, State Key Lab of CAD&CG, Department of Mathematics, Zhejiang University, Hangzhou, China. E-mail: ywguo@cad.zju.edu.cn Contract/grant sponsor: 973 Program of China; contract/grant number: 2002CB312101. Contract/grant sponsor: NSFC; contract/grant numbers: 60033010; 60403038
computer animation Y.GUO ET AL. virtual worlds 36 106 141 175 Figure 1.An example of our video retexturing.The top row lists 6 frames selected from a video clip with 175 frames,and the bottom row shows their corresponding retexturing results. each concerned pixel and making use of its Y compo- the screen,it must undergo a projective transformation. nent which encodes the brightness information of each The resultant image is therefore a non-trivial mapping pixel on the interest region. of the 2D manifold of the original surface depending on As for video retexturing,the main idea is to retexture the shape of the surface.The problem becomes harder a user specified key frame using the image retexturing for retexturing because the 3D shape of the underlying algorithm,and then propagate iteratively the replaced surface is unknown.To simulate the non-linear map- texture adhering to the key frame onto other frames. ping,Liu et al.3 introduced a user-assisted adjustment Here,two key issues for video retexturing need to be on the regular grid of the real texture,and obtained a addressed carefully:one is the texture drifting problem bijective mapping between the regular grid of the tex- among frame sequences,and the other is the visibility ture and the deformed grid of the surface image.Ob- shift between adjacent frames.We introduce a tracking viously,this method requires elaborate user interaction algorithm of feature-points coupled with a mesh based and is only suitable to regular textures.Assuming that optimization scheme to resolve the texture drifting the lighting satisfies Lambertian reflectance model,Fang problem efficiently.Meanwhile,graph cut algorithm is et al.recovered the geometry of the specified area using applied to a trimap along the interest region boundary SFS approximation and derived a propagation rule to to handle the case of visibility shift.Figure 1 demon- recalculate the mapping between the surface image and strates an example of using our video retexturing algo- the new texture. rithm on a video sequence with 175 frames. Extracting lighting information from real images is The remainder of this paper is organized as follows: another challenge for retexturing,Tsin et al.suggested a Section Related Work'presents a brief overview of Bayesian framework based on certain lighting distribu- related previous work.Section Image Retexturing'de- tion model,which relies on the color observation at each scribes the image retexturing of our approach including pixel.Oh et al.presented an algorithm for decoupling mesh generation,texture-coordinates calculation,light- texture illuminance from the image by applying an ing effects and experimental results.Section Video image processing filter.They assumed that large scale Retexturing'further addresses video retexturing using luminance variations are due to the lighting,while small motion tracking and the graph cut algorithms,as well as scale details are due to the texture.Welsh et al.pro- the video retexturing results.Finally,the summary and posed a texture synthesis like algorithm for transferring future research is given in Section 'Conclusions and color into grayscale image,the algorithm works on the Future Work' laB space and transfers a,B components from the sam- ple color image to the grayscale image.Its results auto- matically preserve the lighting effect of the original Related Work grayscale image. Keeping good track of moving objects in video is a Texture mapping needs to set a correspondence be- common goal in the vision and video editing field. tween each point on the 2D texture image and that on Those pixel-wise,non-parametric algorithms (i.e.,opti- the specified 3D surface.When a surface is displayed on cal flow)are robust to small-scale motion only.For Copyright C 2005 John Wiley Sons,Ltd. 452 Comp.Anim.Virtual Worlds 2005;16:451-461
each concerned pixel and making use of its Y component which encodes the brightness information of each pixel on the interest region. As for video retexturing, the main idea is to retexture a user specified key frame using the image retexturing algorithm, and then propagate iteratively the replaced texture adhering to the key frame onto other frames. Here, two key issues for video retexturing need to be addressed carefully: one is the texture drifting problem among frame sequences, and the other is the visibility shift between adjacent frames. We introduce a tracking algorithm of feature-points coupled with a mesh based optimization scheme to resolve the texture drifting problem efficiently. Meanwhile, graph cut algorithm is applied to a trimap along the interest region boundary to handle the case of visibility shift. Figure 1 demonstrates an example of using our video retexturing algorithm on a video sequence with 175 frames. The remainder of this paper is organized as follows: Section ‘Related Work’ presents a brief overview of related previous work. Section ‘Image Retexturing’ describes the image retexturing of our approach including mesh generation, texture-coordinates calculation, lighting effects and experimental results. Section ‘Video Retexturing’ further addresses video retexturing using motion tracking and the graph cut algorithms, as well as the video retexturing results. Finally, the summary and future research is given in Section ‘Conclusions and Future Work’. Related Work Texture mapping needs to set a correspondence between each point on the 2D texture image and that on the specified 3D surface. When a surface is displayed on the screen, it must undergo a projective transformation. The resultant image is therefore a non-trivial mapping of the 2D manifold of the original surface depending on the shape of the surface. The problem becomes harder for retexturing because the 3D shape of the underlying surface is unknown. To simulate the non-linear mapping, Liu et al.3 introduced a user-assisted adjustment on the regular grid of the real texture, and obtained a bijective mapping between the regular grid of the texture and the deformed grid of the surface image. Obviously, this method requires elaborate user interaction and is only suitable to regular textures. Assuming that the lighting satisfies Lambertian reflectance model, Fang et al.5 recovered the geometry of the specified area using SFS approximation and derived a propagation rule to recalculate the mapping between the surface image and the new texture. Extracting lighting information from real images is another challenge for retexturing, Tsin et al. 2 suggested a Bayesian framework based on certain lighting distribution model, which relies on the color observation at each pixel. Oh et al. 6 presented an algorithm for decoupling texture illuminance from the image by applying an image processing filter. They assumed that large scale luminance variations are due to the lighting, while small scale details are due to the texture. Welsh et al. 7 proposed a texture synthesis like algorithm for transferring color into grayscale image, the algorithm works on the l space and transfers ; components from the sample color image to the grayscale image. Its results automatically preserve the lighting effect of the original grayscale image. Keeping good track of moving objects in video is a common goal in the vision and video editing field. Those pixel-wise, non-parametric algorithms (i.e., optical flow8 ) are robust to small-scale motion only. For Figure 1. An example of our video retexturing. The top row lists 6 frames selected from a video clip with 175 frames, and the bottom row shows their corresponding retexturing results. Y. GUO ET AL. ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 452 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation virtual worlds IMAGE AND VIDEO RETEXTURING large-scale motions,tracking methods based on feature Texture Coordinates Calculation points and parametric models are more preferable. Feature based tracking can capture the motion of rota- The mapping from the new texture to the concerned tion,scaling etc.Tracking features with underlying region should be non-linear,to account for the distor- model can further reduce the risk of error,for example, tion effect of the replaced texture induced by the under- Jin et al.10 used a combined model of geometry and pho- lying geometry.As reconstructing the geometry of the tometry to track features and detect outliers in video. underlying surface with SFS and then performing tex- Visibility change may cause problems in tracking,as ture mapping or synthesis can be unstable and costly, new part may appear and old part may disappear in a we calculate the texture coordinates for each pixel video sequence.Both Agarwala et aland Wang et al within the interest region directly by solving an energy introduced an interpolation based,user assisted contour minimization problem. tracking framework for tracking interested part in video For the further description,we use the following sequences.Chuang et al.13 described a video-matting notations.We use I(x,y)to denote the color intensity algorithm based on accurate tracking of the specified of a pixel (x,y)on the image,VI(x,y)=(I,I)to repre- trimap.A trimap is a labeling image for which 0 stands sent the color gradient at (x,y),I=I(x,y)-I(x-1,y) for background,1 stands for foreground and the rest is for the horizontal component of the gradient,and the unknown region to be labeled. Iv=I(x,y)-I(x,y-1)for the vertical component. Suppose that a new texture with adequate size is first laid on the concerned region without distortion.In this Image Retexturing case,the initial texture coordinate for the pixel (x,y) within the concerned region is (uo(x,y),vo(x,y)).Similar In this section,we present a novel approach for image to conventional methods,we use (u(x,y),v(x,y))to retexturing.Assume that a new texture with adequate specify the final texture coordinates incurred by the size is given,as discussed above,the key issue here lies texture distortion. in how to construct a mapping from the new texture Our algorithm for computing the final texture coordi- domain to the concerned region on the original image. nates is based on the assumption that,in the intensity To achieve this,we first generate an initial 2D mesh on field of the image,the depth variation of the local the concerned region and let its shape conform with the surface is considered proportional to the local gradient underlying geometry of this region on the original transition.In fact the mapping between a point on the image. new texture domain and a pixel within the concerned region is determined by concatenate transform.That is, Mesh Generation the texture coordinates of adjacent pixels are inter- related,and there exists an offset between them.Actu- Generating a proper initial mesh for video tracking has ally,this offset can be conducted via the underlying been addressed in the field of video compensation for local geometry. compression.In Reference [14],nodes of the mesh are Figure 2 illustrates the calculation of offset in 1D case. first extracted based on the image features,such as the Let x and x-1 be two adjacent points,then according to spatial gradient,the displaced frame difference(DFD). above assumption,the variation of their underlying A mesh is then built with these nodes,using the con- depths can be written as h(x)-h(x-1)=k.VI(x), strained Delaunay triangulation. where VI(x)is the image gradient at the position x in Here we propose a semi-automatic algorithm ac- 1D case,and k is the proportion derived from the counting for both the image feature of edges and gra- assumption.This follows that the offset for the texture dients.It performs in three steps.The user first coordinates between x and x-1 should be the length of interactively outlines a boundary along the interested the green line in Figure 2: region using snakes.Then,the standard edge detection operator,for example,Canny operator,is applied inside the confined region,so some initial nodes are automa- V1+(k.7(x)2 tically generated on these detected edges.Other points can be introduced by the user when necessary.Delau- nay triangulation algorithm is finally applied to yield an Similarly,as for the 2D case of texture coordinates, initial mesh M over the region of interest. there exists the offsets for the texture coordinates Copyright C 2005 John Wiley Sons,Ltd. 453 Comp.Anim.Virtual Worlds 2005;16:451-461
large-scale motions, tracking methods based on feature points and parametric models are more preferable. Feature based tracking can capture the motion of rotation, scaling etc.9 Tracking features with underlying model can further reduce the risk of error, for example, Jin et al.10 used a combined model of geometry and photometry to track features and detect outliers in video. Visibility change may cause problems in tracking, as new part may appear and old part may disappear in a video sequence. Both Agarwala et al.11 and Wang et al.12 introduced an interpolation based, user assisted contour tracking framework for tracking interested part in video sequences. Chuang et al.13 described a video-matting algorithm based on accurate tracking of the specified trimap. A trimap is a labeling image for which 0 stands for background, 1 stands for foreground and the rest is the unknown region to be labeled. Image Retexturing In this section, we present a novel approach for image retexturing. Assume that a new texture with adequate size is given, as discussed above, the key issue here lies in how to construct a mapping from the new texture domain to the concerned region on the original image. To achieve this, we first generate an initial 2D mesh on the concerned region and let its shape conform with the underlying geometry of this region on the original image. Mesh Generation Generating a proper initial mesh for video tracking has been addressed in the field of video compensation for compression. In Reference [14], nodes of the mesh are first extracted based on the image features, such as the spatial gradient, the displaced frame difference (DFD). A mesh is then built with these nodes, using the constrained Delaunay triangulation. Here we propose a semi-automatic algorithm accounting for both the image feature of edges and gradients. It performs in three steps. The user first interactively outlines a boundary along the interested region using snakes. Then, the standard edge detection operator, for example, Canny operator, is applied inside the confined region, so some initial nodes are automatically generated on these detected edges. Other points can be introduced by the user when necessary. Delaunay triangulation algorithm is finally applied to yield an initial mesh M over the region of interest. Texture Coordinates Calculation The mapping from the new texture to the concerned region should be non-linear, to account for the distortion effect of the replaced texture induced by the underlying geometry. As reconstructing the geometry of the underlying surface with SFS and then performing texture mapping or synthesis can be unstable and costly, we calculate the texture coordinates for each pixel within the interest region directly by solving an energy minimization problem. For the further description, we use the following notations. We use Iðx; yÞ to denote the color intensity of a pixel ðx; yÞ on the image, rIðx; yÞ¼ðIx;IyÞ to represent the color gradient at ðx; yÞ, Ix ¼ Iðx; yÞ Iðx 1; yÞ for the horizontal component of the gradient, and Iy ¼ Iðx; yÞ Iðx; y 1Þ for the vertical component. Suppose that a new texture with adequate size is first laid on the concerned region without distortion. In this case, the initial texture coordinate for the pixel ðx; yÞ within the concerned region is ðu0ðx; yÞ; v0ðx; yÞÞ. Similar to conventional methods, we use ðuðx; yÞ; vðx; yÞÞ to specify the final texture coordinates incurred by the texture distortion. Our algorithm for computing the final texture coordinates is based on the assumption that, in the intensity field of the image, the depth variation of the local surface is considered proportional to the local gradient transition. In fact the mapping between a point on the new texture domain and a pixel within the concerned region is determined by concatenate transform. That is, the texture coordinates of adjacent pixels are interrelated, and there exists an offset between them. Actually, this offset can be conducted via the underlying local geometry. Figure 2 illustrates the calculation of offset in 1D case. Let x and x 1 be two adjacent points, then according to above assumption, the variation of their underlying depths can be written as hðxÞ hðx 1Þ ¼ k rIðxÞ, where rIðxÞ is the image gradient at the position x in 1D case, and k is the proportion derived from the assumption. This follows that the offset for the texture coordinates between x and x 1 should be the length of the green line in Figure 2: ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ð Þ k rIðxÞ 2 q ð1Þ Similarly, as for the 2D case of texture coordinates, there exists the offsets for the texture coordinates IMAGE AND VIDEO RETEXTURING ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 453 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation Y.GUO ET AL. virtual worlds Surface However,for the pixels lying in the triangles of M, directly application of the offset equations (2)-(4)to them may result in a wried mapping.To reduce the 园 error,we obtain their texture coordinates by solving the following energy minimization problem with respect to the u components (We compute their v components Texture coordinate offset similarly): h(x) k-(x) hx-1) minu(xy） IVu(x,y)-D.(x.y)2 (8) JM here Vu(x,y)=(u(x,y)-u(x-1,y),u(x,y)-u(x,y-1)), and D.(x.y)=(V+(k)2.V1+(kz). Minimizing equation (7),it can be easily converted x- Image coordinate into a set of Poisson equations with the form: Figure 2.Texture coordinate offset for adjacent pixels in 1D △u(x,y)=divDu(x,y) (9) case. in which A,div represent the Laplacian and divergence operator separately.The boundary conditions for above between the pixel (x,y)and its neighbors (x-1,y)and Poisson equations are determined by the u components (x,y-1),which can be expressed as follows: of the texture coordinates of those pixels lying on the edges of M,which are calculated using Equation (7).We u(x,0）-u(x-1,y0=V1+(·1)月 (2) adopt the conjugate gradients algorithm to solve them and it runs very fast. As discussed above,the non-linear mapping between ux,y)-ux,y-1)=V1+k·) (3) the point on the new texture domain and the pixel within the concerned region has been converted into solving a set of linear equations.Although some approx- x,y0)-(x,y-1)=V1+(k·1y)1 (4) imation is assumed,the presented algorithm is trivial to implement and presents satisfactory effects for most of our experiments. (x,0-(x-1,)=V1+(k·1x)2 (5) Lighting Effects where,both ki and k2 are the proportions derived from the assumption.The horizontal component Ix of the As we have obtained the non-linear correspondence image gradient impacts more greatly on the u offset between each pixel within the concerned region and than on the v offset,whereas the vertical component ly that on the new texture,the next step is to map the new impacts more greatly on the v offset than on the u offset, texture while preserving the lighting information en- so we set k and k2 with different values 0.6 and 0.3, coded in the original image.Normally,the intensity of a respectively in our experiments. texture can be regarded as the accumulated effect of the For the pixel lying on the edges of the generated mesh color and brightness.If the brightness information of the M,consider the u component of its texture coordinate, original texture can be extracted independently,fusing which exists: it with the new texture will resolve the problem of preserving the lighting.Fortunately,the YCbCr color u(x,y)lxy)eom u(x-1.y)+1+(ki1)2 (6) space illuminates us. YCbCr is a well-known color space compliant to the Making an approximation that u(x-1,y)=uo(x-1,y), digital video standard,where CbCr mainly represents and considering uo(x-1,y)=uo(x,y)-1,the above the hue of each textured pixel and Y component encodes equation is transformed into: its brightness.We simply copy the CbCr components of the new texture to the target image at each concerned u(x,y)lxsbeom=uo(x,y)+V1+(k1)2-1 (7) pixel during texture mapping,and use a weighted blending of the Y component of both the displayed Copyright C 2005 John Wiley Sons,Ltd. 454 Comp.Anim.Virtual Worlds 2005;16:451-461
between the pixel ðx; yÞ and its neighbors ðx 1; yÞ and ðx; y 1Þ, which can be expressed as follows: uðx; yÞ uðx 1; yÞ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk1 IxÞ 2 q ð2Þ uðx; yÞ uðx; y 1Þ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk2 IyÞ 2 q ð3Þ vðx; yÞ vðx; y 1Þ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk1 IyÞ 2 q ð4Þ vðx; yÞ vðx 1; yÞ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk2 IxÞ 2 q ð5Þ where, both k1 and k2 are the proportions derived from the assumption. The horizontal component Ix of the image gradient impacts more greatly on the u offset than on the v offset, whereas the vertical component Iy impacts more greatly on the v offset than on the u offset, so we set k1 and k2 with different values 0.6 and 0.3, respectively in our experiments. For the pixel lying on the edges of the generated mesh M, consider the u component of its texture coordinate, which exists: uðx; yÞjðx;yÞ2@M ¼ uðx 1; yÞ þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk1 IxÞ 2 q ð6Þ Making an approximation that uðx 1; yÞ ¼ u0ðx 1; yÞ, and considering u0ðx 1; yÞ ¼ u0ðx; yÞ 1, the above equation is transformed into: uðx; yÞjðx;yÞ2@M ¼ u0ðx; yÞ þ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk1 IxÞ 2 q 1 ð7Þ However, for the pixels lying in the triangles of M, directly application of the offset equations (2)–(4) to them may result in a wried mapping. To reduce the error, we obtain their texture coordinates by solving the following energy minimization problem with respect to the u components (We compute their v components similarly): minuðx;yÞ Z M jruðx; yÞ Duðx; yÞj2 ð8Þ here ruðx; yÞ¼ðuðx; yÞuðx 1; yÞ, uðx; yÞ uðx; y 1ÞÞ, and Duðx; yÞ¼ð ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk1 IxÞ 2 q ; ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ðk2 IyÞ 2 q Þ. Minimizing equation (7), it can be easily converted into a set of Poisson equations with the form: uðx; yÞ ¼ divDuðx; yÞ ð9Þ in which , div represent the Laplacian and divergence operator separately. The boundary conditions for above Poisson equations are determined by the u components of the texture coordinates of those pixels lying on the edges of M, which are calculated using Equation (7). We adopt the conjugate gradients algorithm to solve them and it runs very fast. As discussed above, the non-linear mapping between the point on the new texture domain and the pixel within the concerned region has been converted into solving a set of linear equations. Although some approximation is assumed, the presented algorithm is trivial to implement and presents satisfactory effects for most of our experiments. Lighting Effects As we have obtained the non-linear correspondence between each pixel within the concerned region and that on the new texture, the next step is to map the new texture while preserving the lighting information encoded in the original image. Normally, the intensity of a texture can be regarded as the accumulated effect of the color and brightness. If the brightness information of the original texture can be extracted independently, fusing it with the new texture will resolve the problem of preserving the lighting. Fortunately, the YCbCr color space illuminates us. YCbCr is a well-known color space compliant to the digital video standard, where CbCr mainly represents the hue of each textured pixel and Y component encodes its brightness. We simply copy the CbCr components of the new texture to the target image at each concerned pixel during texture mapping, and use a weighted blending of the Y component of both the displayed Figure 2. Texture coordinate offset for adjacent pixels in 1D case. Y. GUO ET AL. ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 454 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation virtual worlds IMAGE AND VIDEO RETEXTURING intensity of the concerned pixel on the original image Here mr stands for the weight balancing between the and that of the corresponding sample point on the new new texture and the brightness of the concerned pixel on texture plane.Define Yt,Cbr,Crt,Yi,Cbi,Cri and Yr,Cbr the image,the bigger m is,the further the lighting of the Cr,as the corresponding components of the new tex- retextured image resembles that of the original image. ture,intensity of the concerned pixel on the original We empirically value it with 0.6 in our experiment. image,and that of final result,respectively,the new intensity of the concerned pixel can be expressed as: Results of Image Retexturing Cb:=Cbr (10) Figure 3 demonstrates our experimental results of image Cry=Cr (11) retexturing.We can see that the effects preserve the Y,=m:×Y:+(1-mt)×Yi (12) shading information,and meanwhile yield the illusion C) Figure 3.In (a)and (b),Left is the original images,right is the retexturing results. 甲0年年w000●000年年年甲导事年带年年年年 Copyright C2005 John Wiley Sons,Ltd. 455 Comp.Anim.Virtual Worlds 2005;16:451-461
intensity of the concerned pixel on the original image and that of the corresponding sample point on the new texture plane. Define Yt;Cbt;Crt, Yi;Cbi;Cri and Yr; Cbr; Crr as the corresponding components of the new texture, intensity of the concerned pixel on the original image, and that of final result, respectively, the new intensity of the concerned pixel can be expressed as: Cbr ¼ Cbt ð10Þ Crr ¼ Crt ð11Þ Yr ¼ mt Yt þ ð1 mtÞ Yi ð12Þ Here mt stands for the weight balancing between the new texture and the brightness of the concerned pixel on the image, the bigger mt is, the further the lighting of the retextured image resembles that of the original image. We empirically value it with 0:6 in our experiment. Results of Image Retexturing Figure 3 demonstrates our experimental results of image retexturing. We can see that the effects preserve the shading information, and meanwhile yield the illusion Figure 3. In (a) and (b), Left is the original images, right is the retexturing results. IMAGE AND VIDEO RETEXTURING ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 455 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation Y.GUO ET AL. virtual worlds that the new texture adheres to the underlying surface transferred to its corresponding triangle of the tracked of the original image. frame.We calculate the corresponding mesh points in It takes 0.610 seconds,0.896 seconds to retexture (a) the next frame using the optical flow algorithm.8 How- and(b)respectively,in addition to the user interaction, ever,the tracking may not be accurate enough,some under an environment of a Pentium IV 2.4GHz PC with further constraints are thus made on the mesh to keep 1G main memory.Note that the woman body (b)is the coherence of the mesh topology. retextured by several textures with diverse patterns,and There are three major motion tracking inconsistencies each pattern is mapped onto the concerned region to be considered as bad tracking: independently.As our algorithm mainly focuses on the texture distortion induced by the underlying geo- Matching inconsistency.If the color difference between metry,the global orientation of the replaced texture,for a mesh point in the current frame and that in the example,the orientation of the texture on the arm in(b), tracked frame is greater than a threshold,then the is achieved by user intervention. tracking point is considered as a mismatch. Orientation inconsistency.It is caused when any of the orientations of the triangles incident upon a mesh Video Retexturing point is flipped. Relative motion inconsistency.It happens when the As video is composed of a sequence of frames, motion vector of a mesh point differs too much from video retexturing is more complicated than image its neighbors. retexturing If any type of the above inconsistencies is detected for On the one hand,it is unpractical to retexture each a mesh point,we call it an unstable point;otherwise a frame of the video clip separately.A feasible way is to stable point.The position of the unstable point needs to retexture a key frame selected from the video,for be recalculated to compensate for the inconsistency.We example,the first frame or the user-specified frame adopt the algorithm similar to Reference [14]to inter- with maximum visibility,using the algorithm described polate the motion of the point from its neighbors,with in Section 'Image Retexturing',and then to iteratively an inverse distance weight scheme.Rather than includ- propagate the replaced texture adhering to the key ing all the neighboring nodes,we only select those stable frame onto other frames.In this approach,the texture points.Assume po is an unstable point with neighboring coordinate of the same point on adjacent frames must be nodes pi,p2,...Pn,(txi,ti)denotes the motion vector for accurately tracked.Although there are relevant algo- Pi,and di specifies the distance from pi to po,then the rithms in the field of computer vision,most of them are new interpolated motion vector(txo,fo)is expressed as: not robust enough to deal with all cases. On the other hand,the problem of visibility shift must n=台04 be taken into account,which occurs when part of the (13) ∑=1(0/d texture near the boundary shows up or disappears. Bradshaw15 assumed that the motion discontinuity 二1o0/4 modeling can be decoupled from the affine motion tyo= ∑=1a0/d (14) estimation.With the same assumption,we introduce a trimap along the boundary based on mesh constraints, with which is used to track the motion on the boundary more accurately and to efficiently handle the problem of the 1 if pi is an stable point visibility shift. (0)= 10 otherwise In the following section,we describe our approaches for dealing with each of the above problems in detail. The second and third rows of Figure 5 compare the results(the third row)generated by our algorithm with Motion Tracking those (the second row)obtained without mesh optimi- zation.We can see that in the case when the material The generated mesh for the key frame facilitates the of the object is too smooth inside the concerned region, motion tracking.Starting from the key frame,positions the optical flow algorithm fails frequently,while by of the mesh points are tracked frame by frame.The using our mesh constraint algorithm,the errors can be texture within each triangle of the current frame is mostly corrected. Copyright C 2005 John Wiley Sons,Ltd. 456 Comp.Anim.Virtual Worlds 2005;16:451-461
that the new texture adheres to the underlying surface of the original image. It takes 0.610 seconds, 0.896 seconds to retexture (a) and (b) respectively, in addition to the user interaction, under an environment of a Pentium IV 2.4 GHz PC with 1G main memory. Note that the woman body (b) is retextured by several textures with diverse patterns, and each pattern is mapped onto the concerned region independently. As our algorithm mainly focuses on the texture distortion induced by the underlying geometry, the global orientation of the replaced texture, for example, the orientation of the texture on the arm in (b), is achieved by user intervention. Video Retexturing As video is composed of a sequence of frames, video retexturing is more complicated than image retexturing. On the one hand, it is unpractical to retexture each frame of the video clip separately. A feasible way is to retexture a key frame selected from the video, for example, the first frame or the user-specified frame with maximum visibility, using the algorithm described in Section ‘Image Retexturing’, and then to iteratively propagate the replaced texture adhering to the key frame onto other frames. In this approach, the texture coordinate of the same point on adjacent frames must be accurately tracked. Although there are relevant algorithms in the field of computer vision, most of them are not robust enough to deal with all cases. On the other hand, the problem of visibility shift must be taken into account, which occurs when part of the texture near the boundary shows up or disappears. Bradshaw15 assumed that the motion discontinuity modeling can be decoupled from the affine motion estimation. With the same assumption, we introduce a trimap along the boundary based on mesh constraints, which is used to track the motion on the boundary more accurately and to efficiently handle the problem of the visibility shift. In the following section, we describe our approaches for dealing with each of the above problems in detail. MotionTracking The generated mesh for the key frame facilitates the motion tracking. Starting from the key frame, positions of the mesh points are tracked frame by frame. The texture within each triangle of the current frame is transferred to its corresponding triangle of the tracked frame. We calculate the corresponding mesh points in the next frame using the optical flow algorithm.8 However, the tracking may not be accurate enough, some further constraints are thus made on the mesh to keep the coherence of the mesh topology. There are three major motion tracking inconsistencies to be considered as bad tracking: * Matching inconsistency. If the color difference between a mesh point in the current frame and that in the tracked frame is greater than a threshold, then the tracking point is considered as a mismatch. * Orientation inconsistency. It is caused when any of the orientations of the triangles incident upon a mesh point is flipped. * Relative motion inconsistency. It happens when the motion vector of a mesh point differs too much from its neighbors. If any type of the above inconsistencies is detected for a mesh point, we call it an unstable point; otherwise a stable point. The position of the unstable point needs to be recalculated to compensate for the inconsistency. We adopt the algorithm similar to Reference [14] to interpolate the motion of the point from its neighbors, with an inverse distance weight scheme. Rather than including all the neighboring nodes, we only select those stable points. Assume p0 is an unstable point with neighboring nodes p1; p2; ... pn, ðtxi;tyiÞ denotes the motion vector for pi, and di specifies the distance from pi to p0, then the new interpolated motion vector ðtx0;ty0Þ is expressed as: tx0 ¼ Pn i¼1 P ðiÞtxi=di n i¼1 ðiÞ=di ð13Þ ty0 ¼ Pn i¼1 ðiÞtyi=di Pn i¼1 ðiÞ=di ð14Þ with ðiÞ ¼ 1 if pi is an stable point 0 otherwise The second and third rows of Figure 5 compare the results (the third row) generated by our algorithm with those (the second row) obtained without mesh optimization. We can see that in the case when the material of the object is too smooth inside the concerned region, the optical flow algorithm fails frequently, while by using our mesh constraint algorithm, the errors can be mostly corrected. Y. GUO ET AL. ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 456 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation virtual worlds IMAGE AND VIDEO RETEXTURING Visibility Shift respect to the boundary edges.The interior region TF of Tu belongs to the concerned region doubtless, The unstable mesh points might be detected on the whereas the exterior region Ts falls certainly into the boundary of the concerned region,this is often caused background. by the pose changing of the body or camera movement. After constructing automatically the trimap along the For example,if the interest object rotates in,the current boundary,the graph cut algorithm6 is then applied to boundary triangles should shrink in,part of texture extracting the texture part in it.The energy function of shifts from visible to invisible.On the opposite,if the graph cut is endowed with the form defined by?which object rotates out,the boundary triangles extend out and captures the texture feature efficiently.Additionally,the some previously occluded part shifts from invisible to foreground and background model is learnt only once visible,this part must be retextured by fetching a texture in the key frame,therefore,the speed does not slow patch from the boundary of the texture region occupied down. by the key frame on the initial texture plane.So the key Finally,as the texture part is extracted inside the issue is to find the new boundary for the concerned trimap,mesh points are either appended or abandoned region on the current frame.We introduce a trimap with respect to whether there is a triangle inside cover- Referece [13]along the boundary to handle this problem ing 90%of the texture and a remeshing operation is as shown in Figure 4(a).The unknown region Tu of the performed accordingly.Figure 4 illustrates our idea. trimap is a group of triangles which is built by the mirrored reflection of those boundary triangles with Algorithm for Video Retexturing The whole algorithm for retexturing on video is sum- marized as Table 1: Select a key frame and triangulate the concerned region; Perform image retexturing on the key frame; (b) Starting from the key frame,tracking and retext- uring other frames backward and forward: begin Calculate the corresponding mesh points in the subsequent frame using optical flow algorithm; for each obtained mesh point Detect whether it is an unstable point; if true (d) Recalculate its position according to its neighboring points; end end Construct trimap along the boundary and apply graph cut algorithm to obtain the texture part in the trimap;append or delete points if necessary, and remesh the relevant region; Transfer the texture of the previous frame to the (e) (0 current frame: Figure 4.Illustration of trimap.(a)The current frame and Incorporate the shading information encode by the current frame; constructed trimap.(b)A zoomed-in subregion of Tu.(c)&(e) end The objects rotates in and out.(d)&(f)Apply graph cut and perform remeshing. Table l.Algorithm for video retexturing Copyright C 2005 John Wiley Sons,Ltd. 457 Comp.Anim.Virtual Worlds 2005;16:451-461
Visibility Shift The unstable mesh points might be detected on the boundary of the concerned region, this is often caused by the pose changing of the body or camera movement. For example, if the interest object rotates in, the current boundary triangles should shrink in, part of texture shifts from visible to invisible. On the opposite, if the object rotates out, the boundary triangles extend out and some previously occluded part shifts from invisible to visible, this part must be retextured by fetching a texture patch from the boundary of the texture region occupied by the key frame on the initial texture plane. So the key issue is to find the new boundary for the concerned region on the current frame. We introduce a trimap Referece [13] along the boundary to handle this problem as shown in Figure 4(a). The unknown region TU of the trimap is a group of triangles which is built by the mirrored reflection of those boundary triangles with respect to the boundary edges. The interior region TF of TU belongs to the concerned region doubtless, whereas the exterior region TB falls certainly into the background. After constructing automatically the trimap along the boundary, the graph cut algorithm16 is then applied to extracting the texture part in it. The energy function of graph cut is endowed with the form defined by17 which captures the texture feature efficiently. Additionally, the foreground and background model is learnt only once in the key frame, therefore, the speed does not slow down. Finally, as the texture part is extracted inside the trimap, mesh points are either appended or abandoned with respect to whether there is a triangle inside covering 90% of the texture and a remeshing operation is performed accordingly. Figure 4 illustrates our idea. Algorithm for Video Retexturing The whole algorithm for retexturing on video is summarized as Table 1: Figure 4. Illustration of trimap. (a) The current frame and constructed trimap. (b) A zoomed-in subregion of TU. (c) & (e) The objects rotates in and out. (d) & (f) Apply graph cut and perform remeshing. Select a key frame and triangulate the concerned region; Perform image retexturing on the key frame; Starting from the key frame, tracking and retexturing other frames backward and forward: begin Calculate the correspondingmesh pointsin the subsequent frame using optical flow algorithm; for each obtainedmesh point Detect whether itis an unstable point; if true Recalculate its position according to its neighboring points; end end Construct trimap along the boundary and apply graph cut algorithm to obtain the texture partin the trimap; append or delete points if necessary, and remesh the relevant region; Transfer the texture of the previous frame to the current frame; Incorporate the shadinginformation encode by the current frame; end Table1. Algorithm for video retexturing IMAGE AND VIDEO RETEXTURING ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 457 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation Y.GUO ET AL. virtual worlds Note that,for the tracked frame,the final retextured after a few frames.Using our mesh optimization algo- effect should incorporate with the shading information rithm,however,those unstable nodes are detected and encoded by this frame in the light of the algorithm their positions are adjusted. described in subsection 3.3. Figure 6 demonstrates the capability of our approach in dealing with the visibility shift(this video contains Results on Video Retexturing 19 frames),the 14th frame is specified as the key frame.Although the arm of the woman moves signifi- We have performed our experiments on an Intel Pen- cantly in the sequences,our trimap based algorithm tium IV 2.4GHz PC with 1G main memory under the accurately tracked this,the shading effect is also realis- Windows XP operating system.The timing for retextur- tic.See the attached video for the above video retex- ing a video varies from several seconds to less than a turing examples. minute,in addition to the user interaction,according to different video length and area of the concerned region. Figure 5 shows the results of a video clip(123 frames) Conclusions and Future Work with and without mesh constraints,the 9th frame is selected as the key frame.We can see that in the case of We have proposed and developed a novel image/video retexturing a smooth object whose feature is less ob- retexturing approach that preserves the original shad- vious,the optical flow algorithm fails frequently.Some ing effects with the unknown surface geometry and nodes tracked without constraints may wave severely, lighting conditions.The key issue of image retexturing 41 82 123 Figure 5.Part of the frames of our video retexturing results proving the validity of our mesh-based constraints.The second row illustrates the tracked mesh woithout constraints.The third row demonstrates the mesh with mesh-based constraints. Copyright C 2005 John Wiley Sons,Ltd. 458 Comp.Anim.Virtual Worlds 2005;16:451-461
Note that, for the tracked frame, the final retextured effect should incorporate with the shading information encoded by this frame in the light of the algorithm described in subsection 3.3. Results onVideo Retexturing We have performed our experiments on an Intel Pentium IV 2.4 GHz PC with 1G main memory under the Windows XP operating system. The timing for retexturing a video varies from several seconds to less than a minute, in addition to the user interaction, according to different video length and area of the concerned region. Figure 5 shows the results of a video clip (123 frames) with and without mesh constraints, the 9th frame is selected as the key frame. We can see that in the case of retexturing a smooth object whose feature is less obvious, the optical flow algorithm fails frequently. Some nodes tracked without constraints may wave severely, after a few frames. Using our mesh optimization algorithm, however, those unstable nodes are detected and their positions are adjusted. Figure 6 demonstrates the capability of our approach in dealing with the visibility shift (this video contains 19 frames), the 14th frame is specified as the key frame. Although the arm of the woman moves signifi- cantly in the sequences, our trimap based algorithm accurately tracked this, the shading effect is also realistic. See the attached video for the above video retexturing examples. Conclusions and FutureWork We have proposed and developed a novel image/video retexturing approach that preserves the original shading effects with the unknown surface geometry and lighting conditions. The key issue of image retexturing Figure 5. Part of the frames of our video retexturing results proving the validity of our mesh-based constraints. The second row illustrates the tracked mesh without constraints. The third row demonstrates the mesh with mesh-based constraints. Y. GUO ET AL. ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 458 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation virtual worlds IMAGE AND VIDEO RETEXTURING 13 19 Figure 6.Several frames of a video retexturing result with visibility shift (the waving arm).Note that the shadow effect is realistic. is how to construct the non-linear mapping between the ACKNOWLEDGEMENTS new texture plane and the concerned region on the image.We successfully convert this problem into sol- We wish to thank the anonymous reviewers for their valuable ving the linear Poisson equations based on some as- comments.Most of our video clips for experiments are fetched sumptions.Video retexturing is further accomplished from the demo video of the paper.3The graph cut source code through the stable tracking of the mesh points and the is obtained from the website of the author of Reference [18). graph cut algorithm to treat with the visibility shift. This project is supported in partial by 973 Program of China Although initial experiments have shown some en- No.2002CB312101)and NSFC(No.60033010)&No.60403038). couraging results,our approach is not robust enough to handle all cases.The image retexturing mainly focuses on the texture distortion,and user interaction is needed References for the creation of global orientation of the replaced texture.Our method of preserving lightness works for 1.Efros AA,Freeman WT.Image quilting for texture synth- the input images with almost textureless surface,the esis and transfer.In Proceedings of SIGGRAPH 2001,Los method in Reference [3]may provide a more general Angeles,CA,U.S.A.,August 2001,pp.341-346. solution.Our mesh optimization is mainly designed to 2.Tsin Y,Liu Y,Ramesh V.Texture replacement in real images.In IEEE Conference on Computer Vision and Pattern handle large-scale inconsistencies accounting for the Recognition 2001,Kauai Marriott,Hawaii,U.S.A.,July serious mesh distortion,slight waving of new texture 2001,PP.539-544. between adjacent frames still exists in our experiment. 3.Liu YX,Lin WC,Hays J.Near regular texture analysis and The solution is to define more strict threshold value for manipulation.ACM Transactions on Graphics 2004;23(3): unstable nodes detection,or to allow the users to adjust 368-376. 4.Liu YX,Lin WC.Deformable Texture:The Irregular- those unsatisfactory tracked nodes interactively.Be- Regular-Irregular Cycle.Technical Report CMU-RI-TR- sides,our algorithm has not yet solved the problem of 03-26,Robotics Institute,Carnegie Mellon University, occlusion and uncovering completely. Pittsburgh,PA,2003. Future work includes exploring better mesh optimi- 5.Fang H,Hart J.Textureshop:texture synthesis as a photo- zation using harder'constraints,and better treatment of graph editing tool.ACM Transactions on Graphics 2004; 23(3):354-359. occlusion and uncovering problems.Graphics hardware 6.OH BM,Chen M,Dorsey J,Durand F.Image-based acceleration can also be embedded in our image/video modeling and photo editing.In Proceedings of SIGGRAPH retexturing approach. 2001,Los Angeles,CA,U.S.A.,August 2001.pp.433-442. Copyright C 2005 John Wiley Sons,Ltd. 459 Comp.Anim.Virtual Worlds 2005;16:451-461
is how to construct the non-linear mapping between the new texture plane and the concerned region on the image. We successfully convert this problem into solving the linear Poisson equations based on some assumptions. Video retexturing is further accomplished through the stable tracking of the mesh points and the graph cut algorithm to treat with the visibility shift. Although initial experiments have shown some encouraging results, our approach is not robust enough to handle all cases. The image retexturing mainly focuses on the texture distortion, and user interaction is needed for the creation of global orientation of the replaced texture. Our method of preserving lightness works for the input images with almost textureless surface, the method in Reference [3] may provide a more general solution. Our mesh optimization is mainly designed to handle large-scale inconsistencies accounting for the serious mesh distortion, slight waving of new texture between adjacent frames still exists in our experiment. The solution is to define more strict threshold value for unstable nodes detection, or to allow the users to adjust those unsatisfactory tracked nodes interactively. Besides, our algorithm has not yet solved the problem of occlusion and uncovering completely. Future work includes exploring better mesh optimization using ‘harder’ constraints, and better treatment of occlusion and uncovering problems. Graphics hardware acceleration can also be embedded in our image/video retexturing approach. ACKNOWLEDGEMENTS We wish to thank the anonymous reviewers for their valuable comments. Most of our video clips for experiments are fetched from the demo video of the paper.13 The graph cut source code is obtained from the website of the author of Reference [18]. This project is supported in partial by 973 Program of China (No.2002CB312101) and NSFC (No. 60033010) & (No.60403038). References 1. Efros AA, Freeman WT. Image quilting for texture synthesis and transfer. In Proceedings of SIGGRAPH 2001, Los Angeles, CA, U.S.A., August 2001, pp. 341–346. 2. Tsin Y, Liu Y, Ramesh V. Texture replacement in real images. In IEEE Conference on Computer Vision and Pattern Recognition 2001, Kauai Marriott, Hawaii, U.S.A., July 2001, pp. 539–544. 3. Liu YX, Lin WC, Hays J. Near regular texture analysis and manipulation. ACM Transactions on Graphics 2004; 23(3): 368–376. 4. Liu YX, Lin WC. Deformable Texture: The IrregularRegular-Irregular Cycle. Technical Report CMU-RI-TR- 03-26, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, 2003. 5. Fang H, Hart J. Textureshop: texture synthesis as a photograph editing tool. ACM Transactions on Graphics 2004; 23(3): 354–359. 6. OH BM, Chen M, Dorsey J, Durand F. Image-based modeling and photo editing. In Proceedings of SIGGRAPH 2001, Los Angeles, CA, U.S.A., August 2001. pp. 433–442. Figure 6. Several frames of a video retexturing result with visibility shift (the waving arm). Note that the shadow effect is realistic. IMAGE AND VIDEO RETEXTURING ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 459 Comp. Anim. Virtual Worlds 2005; 16: 451–461
computer animation Y.GUO ET AL. virtual worlds 7.Welsh T,Ashikhmin M,Mueller K.Transferring color to Authors'biographies: greyscale images.ACM Transactions on Gaphics 2002; 21(3277-280. 8.Beauchemin SS,Barron JL.The Computation of Optical Flow. ACM Computing Surveys 1995;27(3):433-467. 9.Shi JB,Tomasi C.Good features to track.In IEEE Conference on Computer Vision and Pattern Recognition(CVPR'94),Seat- tle,U.S.A,June1994,Pp.593-600. 10.Jin HL,Favaro P,Soatto S.Real-time feature tracking and outlier rejection with changes in illumination.In The Eighth IEEE International Conference on Computer Vision,Vancou- ver,Canada,July 2001,pp.684-689. 11.Agarwala A,Hertzmann A,Salesin DH,Seitz1 SM.Key- Yanwen Guo is a Ph.D candidate in State Key Lab of frame-based tracking for rotoscoping and animation. CAD&CG,Zhejiang University,China.Now his re- ACM Transactions on Graphics 2004;23(3):584-591. search interests include geometry processing,video- 12.Wang J,Xu YQ,Shum HY,Cohen MF.Video tooning. based rendering,computer vision,etc. ACM Transactions on Gaphics 2004;23(3):574-583. 13.Chuang YY,Agarwala A,Curless B,Salesin DH,SzeliskiR. Video matting of complex scenes.ACM Transactions on Gaphics2002;21(3:243-248. 14.Altunbasak Y,Tekalp AM.Closed-form connectivity-pre- serving solutions for motion compensation using 2-D meshes.IEEE Transactions on Image Processing 1997;6(9): 1255-1269. 15.Bradshaw DB.Motion estimation and compensation of video sequences using affine transforms.PhD thesis,University of Jin Wang is a researcher in State Key Lab of CAD&CG. Cambridge,1999. Now his research interests include computer graphics 16.Boykov Y,Jolly MP.Interactive graph cuts for optimal computer vision. boundary and region segmentation of objects in n-d images.In The Eighth IEEE International Conference on Com- puter Vision,Vancouver,Canada,July 2001,pp.105-112. 17.Li Y,Sun J,Tang CK,Shum HY.Lazy snapping.ACM Transactions on Gaphics 2004;23(3):303-308. 18.Boykov Y,Kolmogorov V.An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision.IEEE Transactions on Pattern Analysis and Machine Intelligence2004:26(9):1124-1137. Xiang Zeng is a graduate student in Zhejiang Univer- sity.He received his B.E.degree in Computer Science in 2004 from Zhejiang University.His research interests include vision-based graphics and computational com- plexity. Zhongyi Xie received his B.E.degree in Computer Science in June 2005 from Zhejiang University.His research interest is geometry processing. Copyright C 2005 John Wiley Sons,Ltd. 460 Comp.Anim.Virtual Worlds 2005;16:451-461
7. Welsh T, Ashikhmin M, Mueller K. Transferring color to greyscale images. ACM Transactions on Gaphics 2002; 21(3): 277–280. 8. Beauchemin SS, Barron JL. The Computation of Optical Flow. ACM Computing Surveys 1995; 27(3): 433–467. 9. Shi JB, Tomasi C. Good features to track. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’94), Seattle, U.S.A., June 1994, pp. 593–600. 10. Jin HL, Favaro P, Soatto S. Real-time feature tracking and outlier rejection with changes in illumination. In The Eighth IEEE International Conference on Computer Vision, Vancouver, Canada, July 2001, pp. 684–689. 11. Agarwala A, Hertzmann A, Salesin DH, Seitz1 SM. Keyframe-based tracking for rotoscoping and animation. ACM Transactions on Graphics 2004; 23(3): 584–591. 12. Wang J, Xu YQ, Shum HY, Cohen MF. Video tooning. ACM Transactions on Gaphics 2004; 23(3): 574–583. 13. Chuang YY, Agarwala A, Curless B, Salesin DH, Szeliski R. Video matting of complex scenes. ACM Transactions on Gaphics 2002; 21(3): 243–248. 14. Altunbasak Y, Tekalp AM. Closed-form connectivity-preserving solutions for motion compensation using 2-D meshes. IEEE Transactions on Image Processing 1997; 6(9): 1255–1269. 15. Bradshaw DB. Motion estimation and compensation of video sequences using affine transforms. PhD thesis, University of Cambridge, 1999. 16. Boykov Y, Jolly MP. Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In The Eighth IEEE International Conference on Computer Vision, Vancouver, Canada, July 2001, pp. 105–112. 17. Li Y, Sun J, Tang CK, Shum HY. Lazy snapping. ACM Transactions on Gaphics 2004; 23(3): 303–308. 18. Boykov Y, Kolmogorov V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 2004; 26(9): 1124–1137. Authors’ biographies: Yanwen Guo is a Ph.D candidate in State Key Lab of CAD&CG, Zhejiang University, China. Now his research interests include geometry processing, videobased rendering, computer vision, etc. Jin Wang is a researcher in State Key Lab of CAD&CG. Now his research interests include computer graphics & computer vision. Xiang Zeng is a graduate student in Zhejiang University. He received his B.E. degree in Computer Science in 2004 from Zhejiang University. His research interests include vision-based graphics and computational complexity. Zhongyi Xie received his B.E. degree in Computer Science in June 2005 from Zhejiang University. His research interest is geometry processing. Y. GUO ET AL. ****************************************************************************************************** ****************************************************************************************************** Copyright # 2005 John Wiley & Sons, Ltd. 460 Comp. Anim. Virtual Worlds 2005; 16: 451–461