178 Y.Guo et al.Computers Graphics 38(2014)174-182 Fig.6.Viewpoint adjustment for the images with standard cuboid structures.The 1st row is the inputs with cuboid structures marked by blue segments and line constraints by green ones.The 2nd row shows corresponding results with constrained lines,and the results without constraints are given in the 3rd row.(For interpretation of the references to color in this figure caption,the reader is referred to the web version of this article.) Line constraint:Strong edges such as straight lines are impor- Border constraint:Physically each side of image borders should tant visual features.They are vital clues for understanding image be constrained to remain straight.The energy term Es of this content,and should be maintained as-rigid-as possible.We detect constraint is defined similarly to the line constraint. those line segments using Hough transform.Users can also specify Total energy:In summary,combining all the above energy some additional curved edges.Points sampled from the edges and terms,we wish to minimize the following energy function: their connectivity yielded from the corresponding edges are fed arg max AsEs+ALEL+AvEy+4BEB, into the triangulation process beforehand.Let (vi,vi,vk)denote a triplet of vertices on a straight line.To preserve the shape of strong (14) edges,we preserve the length ratio ry of vivk to vivj.and the angle s.t.Fc(Vel...Vek)=0. 0 formed by vivj and vivg in each triplet locally [15].We express where is.i,Ay,and ig are the coefficients weighting different the energy term regarding line constraint as energy terms.Straight and important curved lines as visually E=∑Iwk-9-·R·(y-2, prominent features should be kept.To mimic hard constraints, (11) AL.the weight of straight line constraint,is often set to a bigger value compared with the weight of shape constraint.ly,the with weight of vertical and horizontal line constraint,can be set by COS j the user with respect to image content.Obviously.Ev can be Rj= sin 0j cos 0j (12) enforced as a hard constraint with a bigger iv.It is useful to straighten up those slanted man-made structures in an input Besides the lines detected by Hough transform or specified by image to improve its perceptual quality.In practice,to deal with the user,we use line constraint to preserve the shapes of those possible confliction between border constraint and constraint on salient objects that lie across two different faces of the latent the cuboid structure near image border,Eg often takes effect as a cuboid or the cuboid and the rest image region.A line segment is soft constraint by setting ig to a small value.For all the results in specified on each of such objects,and is fed into E for avoiding this paper we use weights of is=1.=100.v and is are set to heavy distortion. 100 as well if Ey and Eg are taken as hard constraints,and are set to Vertical and horizontal line constraint:Photos taken by amateur 10 for soft constraints. photographers often contain slanted vertical or horizontal lines In is noted that we do not impose the constraint for avoiding due to improper camera rotations,for instance slanted buildings, mesh flip-over in the above energy function.In all our experi- windows,and picture frames.This may cause visual discomfort ments,mesh flipping is seldom encountered.We check it after the when we look at such photos.Our system supports automatic deformed mesh is obtained,and once flipping is detected,we correction of slanted line segments when viewpoint is changed correct them locally. and image is warped.For the slanted cuboid structure,the idea is The energy function is a quadratic function of V.The solution to re-project it properly.The new viewpoint is computed auto- can be obtained efficiently by solving a sparse linear system. matically by letting the projected edges to be horizontal or vertical.While for the rest slanted line segments,this is treated as the vertical and horizontal line constraint.Let I denote a slanted 5.Experiments line segment which can be detected automatically or specified by the user,and (vn,...,vim}represent vertices on I.The vertical line We have implemented our view manipulation algorithm on a constraint is expressed as PC with Intel Core i3-2100 CPU at 3.1 GHz,and experimented with our technique on a variety of images.Some representative results Ev =( (13) are shown in Figs.6-10 and 12. Figs.6 and 7 demonstrate the results on several images of man- The horizontal line constraint is defined similarly. made buildings.The first and third rows are the input imagesLine constraint: Strong edges such as straight lines are impor￾tant visual features. They are vital clues for understanding image content, and should be maintained as-rigid-as possible. We detect those line segments using Hough transform. Users can also specify some additional curved edges. Points sampled from the edges and their connectivity yielded from the corresponding edges are fed into the triangulation process beforehand. Let 〈vi; vj; vk〉 denote a triplet of vertices on a straight line. To preserve the shape of strong edges, we preserve the length ratio rj of vjvk to vivj, and the angle θj formed by vivj and vjvk in each triplet locally [15]. We express the energy term regarding line constraint as EL ¼ ∑ 〈vi;vj;vk〉 ‖ðv′ k v′ j Þrj  Rj  ðv′ j v′ i Þ‖2; ð11Þ with Rj ¼ cos θj  sin θj sin θj cos θj !: ð12Þ Besides the lines detected by Hough transform or specified by the user, we use line constraint to preserve the shapes of those salient objects that lie across two different faces of the latent cuboid or the cuboid and the rest image region. A line segment is specified on each of such objects, and is fed into EL for avoiding heavy distortion. Vertical and horizontal line constraint: Photos taken by amateur photographers often contain slanted vertical or horizontal lines due to improper camera rotations, for instance slanted buildings, windows, and picture frames. This may cause visual discomfort when we look at such photos. Our system supports automatic correction of slanted line segments when viewpoint is changed and image is warped. For the slanted cuboid structure, the idea is to re-project it properly. The new viewpoint is computed auto￾matically by letting the projected edges to be horizontal or vertical. While for the rest slanted line segments, this is treated as the vertical and horizontal line constraint. Let l denote a slanted line segment which can be detected automatically or specified by the user, and fvl1;…; vlmg represent vertices on l. The vertical line constraint is expressed as EV ¼ ∑ l ∑ lm i ¼ 1 ðx′ li x′ l1Þ 2: ð13Þ The horizontal line constraint is defined similarly. Border constraint: Physically each side of image borders should be constrained to remain straight. The energy term EB of this constraint is defined similarly to the line constraint. Total energy: In summary, combining all the above energy terms, we wish to minimize the following energy function: arg max V′ λSES þλLEL þλV EV þλBEB; s:t: FCðv′ c1; …; v′ ckÞ ¼ 0: ð14Þ where λS, λL, λV , and λB are the coefficients weighting different energy terms. Straight and important curved lines as visually prominent features should be kept. To mimic hard constraints, λL, the weight of straight line constraint, is often set to a bigger value compared with the weight of shape constraint. λV , the weight of vertical and horizontal line constraint, can be set by the user with respect to image content. Obviously, EV can be enforced as a hard constraint with a bigger λV . It is useful to straighten up those slanted man-made structures in an input image to improve its perceptual quality. In practice, to deal with possible confliction between border constraint and constraint on the cuboid structure near image border, EB often takes effect as a soft constraint by setting λB to a small value. For all the results in this paper we use weights of λS ¼ 1, λL ¼ 100. λV and λB are set to 100 as well if EV and EB are taken as hard constraints, and are set to 10 for soft constraints. In is noted that we do not impose the constraint for avoiding mesh flip-over in the above energy function. In all our experi￾ments, mesh flipping is seldom encountered. We check it after the deformed mesh is obtained, and once flipping is detected, we correct them locally. The energy function is a quadratic function of V′. The solution can be obtained efficiently by solving a sparse linear system. 5. Experiments We have implemented our view manipulation algorithm on a PC with Intel Core i3-2100 CPU at 3.1 GHz, and experimented with our technique on a variety of images. Some representative results are shown in Figs. 6–10 and 12. Figs. 6 and 7 demonstrate the results on several images of man￾made buildings. The first and third rows are the input images. Fig. 6. Viewpoint adjustment for the images with standard cuboid structures. The 1st row is the inputs with cuboid structures marked by blue segments and line constraints by green ones. The 2nd row shows corresponding results with constrained lines, and the results without constraints are given in the 3rd row. (For interpretation of the references to color in this figure caption, the reader is referred to the web version of this article.) 178 Y. Guo et al. / Computers & Graphics 38 (2014) 174–182