Y.W.Guo et al./Improving Photo Composition Elegantly:Considering Image Similarity During Composition Optimization (a) (b) (c) Figure 1:The image (b)with improved composition generated by our method for an input image (a).(c)shows the result of Liu et al. [LCWCO10].The result exhibits obvious distortion in the cloud region.Moreover,the scale operation they adopt may cause blurring artifacts, see the girl sitting in the wheelchair. however rarely consider this problem.As a result,the sudden The remainder of the paper is organized as follows.The and significant changes in image content may lead to an un- related work is introduced in Section 2.Section 3 describes pleasant user experience even though the optimal aesthetics our objective model that combines the composition aesthet- is achieved.Another limitation of previous methods is that ics and image similarity.Section 4 shows how to compute recomposing the prominent object to the optimal position the recomposed image maximizing the objective model.We suggested by aesthetic assessment directly,without any con- conduct experiments and compare with previous methods in straint on the result,may incur inevitable visual distortion, Section 5,and Section 6 concludes the whole paper. since inherently enhancing composition and reducing dis- tortion conflict with each other,especially for those images 2.Related Work with complex structures.As shown in Figure 1,the result(c) produced by the method in [LCWCO10]exhibits obvious Photo quality assessment and enhancement,as an important distortion in the cloud region,and differs from the original aspect of computational photography,has attracted a large image too much.This may be unacceptable to users body of research.Those works on noise removal,brightness adjustment,and deblur are beyond the scope of this paper. In this paper,we present a new algorithm for improving We mainly review here the relevant methods on assessment the composition aesthetics of an input image while avoiding and enhancement of composition aesthetics.Image retarget- making significant changes to the visual appearance.Visual ing,as an important means for improving composition,is similarity between the optimized image and the original one briefly introduced. is taken into account during the optimization of composition. The similarity is quantized by a measure of structural sim- ilarity called SSIM.We further incorporate a term of edge 2.1.Photo Composition Assessment and Enhancement similarity into the similarity measure in order to reinforce Ke et al.[KTJ06]propose a principled method to assess the preservation of strong edges in images which are impor- photo quality.High level semantic features are designed for tant visual cues.Our objective model combines the aesthetic measuring the perceptual differences between high qual- measurement and the similarity.To compute the optimal im- ity professional photos and low quality snapshots.Differ- age balancing composition aesthetics and visual similarity, ent people.even for the professional photographers may we basically use seam carving to carve out a series of less have different aesthetical criteria in mind when taking and noticeable seams and,correspondingly,to insert the same examining photographs.To bridge the gap between visual number of seams on the image.The optimal image is gener- features and users'evaluation over quality,Bhattacharya et ated by searching the maximum of the objective model dur- al.[BSS10]formulate photo quality evaluation as a machine ing this process.Our method can produce a good quality re- learning problem in which the support vector regressors are composed image in an elegant and user controllable manner. used to learn the mappings from aesthetic features to vi- It is intuitive,easy-to-implement,and runs fast. sual attractiveness on composition.With the same features Our main contribution is a composition optimizing used to evaluate a given composition,the image with poor method which takes into account visual similarity between composition is enhanced by either relocating the segmented the optimized image and the original one.This allows us to foreground onto painted background or balancing the visual produce the composition improved images which have min- weights of different image regions. imal visual distortions,and retain faithful,as much as possi- Liu et al.[LJW10]measure composition aesthetics based ble,to the original image content. on the distributions of detected salient regions and prominent ©2012 The Author()s 2012 The Eurographics Association and Blackwell Publishing Lid.Y. W. Guo et al. / Improving Photo Composition Elegantly: Considering Image Similarity During Composition Optimization (a) (b) (c) Figure 1: The image (b) with improved composition generated by our method for an input image (a). (c) shows the result of Liu et al. [LCWCO10]. The result exhibits obvious distortion in the cloud region. Moreover, the scale operation they adopt may cause blurring artifacts, see the girl sitting in the wheelchair. however rarely consider this problem. As a result, the sudden and significant changes in image content may lead to an un￾pleasant user experience even though the optimal aesthetics is achieved. Another limitation of previous methods is that recomposing the prominent object to the optimal position suggested by aesthetic assessment directly, without any con￾straint on the result, may incur inevitable visual distortion, since inherently enhancing composition and reducing dis￾tortion conflict with each other, especially for those images with complex structures. As shown in Figure 1, the result (c) produced by the method in [LCWCO10] exhibits obvious distortion in the cloud region, and differs from the original image too much. This may be unacceptable to users. In this paper, we present a new algorithm for improving the composition aesthetics of an input image while avoiding making significant changes to the visual appearance. Visual similarity between the optimized image and the original one is taken into account during the optimization of composition. The similarity is quantized by a measure of structural sim￾ilarity called SSIM. We further incorporate a term of edge similarity into the similarity measure in order to reinforce the preservation of strong edges in images which are impor￾tant visual cues. Our objective model combines the aesthetic measurement and the similarity. To compute the optimal im￾age balancing composition aesthetics and visual similarity, we basically use seam carving to carve out a series of less noticeable seams and, correspondingly, to insert the same number of seams on the image. The optimal image is gener￾ated by searching the maximum of the objective model dur￾ing this process. Our method can produce a good quality re￾composed image in an elegant and user controllable manner. It is intuitive, easy-to-implement, and runs fast. Our main contribution is a composition optimizing method which takes into account visual similarity between the optimized image and the original one. This allows us to produce the composition improved images which have min￾imal visual distortions, and retain faithful, as much as possi￾ble, to the original image content. The remainder of the paper is organized as follows. The related work is introduced in Section 2. Section 3 describes our objective model that combines the composition aesthet￾ics and image similarity. Section 4 shows how to compute the recomposed image maximizing the objective model. We conduct experiments and compare with previous methods in Section 5, and Section 6 concludes the whole paper. 2. Related Work Photo quality assessment and enhancement, as an important aspect of computational photography, has attracted a large body of research. Those works on noise removal, brightness adjustment, and deblur are beyond the scope of this paper. We mainly review here the relevant methods on assessment and enhancement of composition aesthetics. Image retarget￾ing, as an important means for improving composition, is briefly introduced. 2.1. Photo Composition Assessment and Enhancement Ke et al. [KTJ06] propose a principled method to assess photo quality. High level semantic features are designed for measuring the perceptual differences between high qual￾ity professional photos and low quality snapshots. Differ￾ent people, even for the professional photographers may have different aesthetical criteria in mind when taking and examining photographs. To bridge the gap between visual features and users’ evaluation over quality, Bhattacharya et al. [BSS10] formulate photo quality evaluation as a machine learning problem in which the support vector regressors are used to learn the mappings from aesthetic features to vi￾sual attractiveness on composition. With the same features used to evaluate a given composition, the image with poor composition is enhanced by either relocating the segmented foreground onto painted background or balancing the visual weights of different image regions. Liu et al. [LJW10] measure composition aesthetics based on the distributions of detected salient regions and prominent
c 2012 The Author(s)
c 2012 The Eurographics Association and Blackwell Publishing Ltd