Y.W.Guo et al./Improving Photo Composition Elegantly:Considering Image Similarity During Composition Optimization can be further accelerated by removing and inserting several seams,rather than only one seam in each step. An advantage of the above heuristic approach that works by successively carving out and inserting a series of seams is that the process the image changes gradually is open to the user.As the seams removed and inserted can be saved with (a) (b) very little extra memory,the user can backtrack to the in- Figure 3:Insertion and removal of seams.(a)For the image with termediate results without re-executing the whole algorithm a monkey as the foreground,the monkey will move right one pixel if once he finds that the final result differs from his input too a seam (yellow)is inserted on its left.The red seam is removed for much. maintaining image size.(b)For the image with two separate sub. jects,seams inserted and removed for adjusting the location of one subject should not sacrifice composition of the other.The left sub- 5.Experiments ject will move right one pixel,and meanwhile,the right will move left one pixel if the yellow seams are inserted and red ones are re- We experimented with our algorithm on a variety of images. moved. Some representative results are shown in Figures 4 and 7. Figure 4 shows that image energies Ee,Es,and E change with the number of seams inserted and removed.Ee in both rows will achieve the maximum if the foreground subjects tions,composition aesthetics is enhanced by inserting and are re-composed onto the optimal positions suggested by removing a certain number of horizontal or (and)vertical the aesthetics rules.However,under the control of similar- seams.Without lose of generality,we illustrate the process ity term Es in objective model,E achieves the maximum by taking horizontal movement of salient objects as an ex- in front of Ee with the increasing number of inserted and ample as shown in Figure 3.That is to say,vertical seams are removed seams.That is to say,foreground subjects in the inserted and removed.Most photographs with distinct fore- images are moved to the positions which are close to,but ground subjects have at most two subjects.For the images not exact,the optimal positions determined by aesthetic op- with an individual subject,it will move toward the target lo- timization. cation by simply removing seams on the side of target loca- tion and inserting seams on the opposite side.It is however a Similar cases are shown in Figure 7 which demonstrates little bit of trouble for the images with two separate subjects. more challenging examples.The images have different reso- For ease of exposition,we call the side of target location the lutions ranging from640×480,878×652to1613×1024 positive side,and opposite side the negative side.To avoid For the images with simple background such as the 6th im- conflict,seams inserted and removed for adjusting the loca- age in the Ist column and the 5th one in the 3rd column,the tion of one object should not sacrifice the composition of the interest objects are moved to the optimal aesthetic positions other object. after composition optimization.However for most photos with complex background structures,the foreground objects We basically adopt the seams suggested by seam carving. are re-located to the new positions near the optimal positions At the beginning,the horizontal and vertical distances be- under the control of the term of image similarity.Our algo- tween positions of subjects and their optimal positions are rithm produces the aesthetically improved images,without computed.The maximum number of seams to be removed noticeable visual distortions and inserted in each direction is then determined.Further- more,the seams are generated by dynamic programming on To demonstrate the effectiveness of our algorithm,we also the original image.By successively carving out a series of compare against the results of previous representative meth- vertical or horizontal seams on the positive side and inserting ods as shown in Figures 1,5,and 6.As shown in Figure new seams on negative side,E(I)will increase at the begin- 5,the results (b)produced by [LJW10]may present obvi- ning as composition aesthetics is enhanced,although image ous distortions in background.See the regions surrounded similarity is reduced.Nevertheless,when a certain amount by the blue rectangles.In contrast,our results(c)avoid such of seams are removed,E(I)reaches the maximum and re- distortions and look visually pleasing.Visual dissimilarity moving more seams will make the image differ from Io too between the results and inputs is penalized by the similar- much.The image version corresponding to the maximum of ity term in our objective model.The comparison in Figure 6 E(I)is the resulting image. shows that our result is comparable to the result of [BSS101. The method of [BSS10]relies on user-guided foreground Computation cost of the above algorithm is mainly con- segmentation and background inpainting for recomposing sumed by the process of computing seams to be removed and the object onto repainted background.The former however inserted.Fortunately,we only need to compute them once on is difficult for the images whose foreground and background the original image.It runs very fast.In addition,the process share similar color appearances.The latter is challenging for ©2012 The Author(s 2012 The Eurographics Association and Blackwell Publishing Lid.Y. W. Guo et al. / Improving Photo Composition Elegantly: Considering Image Similarity During Composition Optimization (a) (b) Figure 3: Insertion and removal of seams. (a) For the image with a monkey as the foreground, the monkey will move right one pixel if a seam (yellow) is inserted on its left. The red seam is removed for maintaining image size. (b) For the image with two separate sub￾jects, seams inserted and removed for adjusting the location of one subject should not sacrifice composition of the other. The left sub￾ject will move right one pixel, and meanwhile, the right will move left one pixel if the yellow seams are inserted and red ones are re￾moved. tions, composition aesthetics is enhanced by inserting and removing a certain number of horizontal or (and) vertical seams. Without lose of generality, we illustrate the process by taking horizontal movement of salient objects as an ex￾ample as shown in Figure 3. That is to say, vertical seams are inserted and removed. Most photographs with distinct fore￾ground subjects have at most two subjects. For the images with an individual subject, it will move toward the target lo￾cation by simply removing seams on the side of target loca￾tion and inserting seams on the opposite side. It is however a little bit of trouble for the images with two separate subjects. For ease of exposition, we call the side of target location the positive side, and opposite side the negative side. To avoid conflict, seams inserted and removed for adjusting the loca￾tion of one object should not sacrifice the composition of the other object. We basically adopt the seams suggested by seam carving. At the beginning, the horizontal and vertical distances be￾tween positions of subjects and their optimal positions are computed. The maximum number of seams to be removed and inserted in each direction is then determined. Further￾more, the seams are generated by dynamic programming on the original image. By successively carving out a series of vertical or horizontal seams on the positive side and inserting new seams on negative side, E(I) will increase at the begin￾ning as composition aesthetics is enhanced, although image similarity is reduced. Nevertheless, when a certain amount of seams are removed, E(I) reaches the maximum and re￾moving more seams will make the image differ from Io too much. The image version corresponding to the maximum of E(I) is the resulting image. Computation cost of the above algorithm is mainly con￾sumed by the process of computing seams to be removed and inserted. Fortunately, we only need to compute them once on the original image. It runs very fast. In addition, the process can be further accelerated by removing and inserting several seams, rather than only one seam in each step. An advantage of the above heuristic approach that works by successively carving out and inserting a series of seams is that the process the image changes gradually is open to the user. As the seams removed and inserted can be saved with very little extra memory, the user can backtrack to the in￾termediate results without re-executing the whole algorithm once he finds that the final result differs from his input too much. 5. Experiments We experimented with our algorithm on a variety of images. Some representative results are shown in Figures 4 and 7. Figure 4 shows that image energies Ee, Es, and E change with the number of seams inserted and removed. Ee in both rows will achieve the maximum if the foreground subjects are re-composed onto the optimal positions suggested by the aesthetics rules. However, under the control of similar￾ity term Es in objective model, E achieves the maximum in front of Ee with the increasing number of inserted and removed seams. That is to say, foreground subjects in the images are moved to the positions which are close to, but not exact, the optimal positions determined by aesthetic op￾timization. Similar cases are shown in Figure 7 which demonstrates more challenging examples. The images have different reso￾lutions ranging from 640 × 480, 878 × 652 to 1613 × 1024. For the images with simple background such as the 6th im￾age in the 1st column and the 5th one in the 3rd column, the interest objects are moved to the optimal aesthetic positions after composition optimization. However for most photos with complex background structures, the foreground objects are re-located to the new positions near the optimal positions under the control of the term of image similarity. Our algo￾rithm produces the aesthetically improved images, without noticeable visual distortions. To demonstrate the effectiveness of our algorithm, we also compare against the results of previous representative meth￾ods as shown in Figures 1, 5, and 6. As shown in Figure 5, the results (b) produced by [LJW10] may present obvi￾ous distortions in background. See the regions surrounded by the blue rectangles. In contrast, our results (c) avoid such distortions and look visually pleasing. Visual dissimilarity between the results and inputs is penalized by the similar￾ity term in our objective model. The comparison in Figure 6 shows that our result is comparable to the result of [BSS10]. The method of [BSS10] relies on user-guided foreground segmentation and background inpainting for recomposing the object onto repainted background. The former however is difficult for the images whose foreground and background share similar color appearances. The latter is challenging for
c 2012 The Author(s)
c 2012 The Eurographics Association and Blackwell Publishing Ltd