正在加载图片...
Y.W.Guo et al.Improving Photo Composition Elegantly:Considering Image Similarity During Composition Optimization uniformly set to 1 [WBSS04].Let ur and mo denote mean 4.Optimization intensity of Ir and Io separately.oIr and o are standard deviations.rlo represents the covariance of image vectors The resulting image is the image maximizing the objective function E(①)， of Ir and Io.()c(,)and s(,are expressed as, Ir =arg maxE (I). (11) I(Ir,Io)= 24r4o+C1 (6) Mir++C1 Previous techniques on photo composition enhancement typically transform the images by image retargeting cou- pled with the cropping operation.We take seam carving 2o1ro10+C2 [AS07]as the basic operation for improving the composi- c(Ir,Io)= (7) i+i。+C2 tion aesthetics.In addition,we assume here two constraints for meeting users'requirements in practice.First,image sizes and aspect ratios should not be altered.Cropping is not allowed in this sense as cropping can lead to the loss of s(Ir,Io)= Irlo +C3 GIrGIo +C3 (8) valuable image information,even though it can prescribe a straightforward solution to optimal re-composition [BSS101. where C,C2.and C3 are constants for avoiding computa- Second,salient regions should be free from zooming in or tional instability.As suggested in [WBSS04].CI and C2 are out since zooming in directly inevitably blurs the subject, set to (KL)2 and (K2L)-in which L is set to 255 for 8-bit while zooming out reduces resolution.With the above con- grayscale images,and Ki and K2 can be set as 0.01 and 0.03 straints,we use a heuristic algorithm to find the solution cor- separately in implementation.C3 is usually set to 2/2 in responding to the optimal image. practice. Seam carving changes image size and aspect ratio by carving out a series of less noticeable seams.A seam is an The essence of SSIM in contrast to tradition metrics is to compare the structures of two images directly.Studies of optimal path of pixels from top to bottom,or left to right defined in terms of local energy.Such a seam can be found cognitive psychology show that human visual perception is using dynamic programming.Note that,in our application to very sensitive to the strong edges in natural images.Since re- targeting techniques such as seam carving [AS07]or mesh- preserve completely the salient objects,local energy defined guided warping [WTSL08.GLS*09]may easily destroy or on salient regions is valued with the maximal value in the deform important edges,it is vital to reinforce edge similar- energy function of seam carving. ity in the similarity measure.To account for this,we adopt We observe that by iteratively carving out seams on one a measure of edge similarity that compares strong edges and side of a salient object and inserting the same number of use it as a compliment to structure comparison in SSIM. seams on the other side simultaneously,image composition can be modified without changing image size.The key prob- Sobel operators with a horizontal edge mask and a vertical lem is thus to determine k,the number of seams to be re- one are first applied to the given image.This yields two edge moved,together with the k seams to be removed and k new maps by exploiting which we can easily compute a gradient seams to be inserted such that the resulting image I maxi- magnitude and an orientation for each edge pixel.An edge mizes E(I).We denote s and{s as the seams orientation histogram with 8 bins in 0-180 can thus be as the candidates for removal and insertion.Each s or s built,and we use it to compute edge similarity as follows, is labelled as 1 if it is selected,and 0 not selected.The opti- e(Ir,Io)=GurHo+C mization is then formulated as a labelling problem which can GHrGHo +C (9) be solved by 0-1 mixed integer programming.Global opti- mization on the parameter space is still computationally ex- where OHr and oHo denote standard deviations of the pensive and may get stuck in local optima easily.We hereby histogram vectors of Ir and Io,separately.HrHo is the develop an efficient heuristic algorithm which finds the solu- covariance of two histograms.Note that,the histogram tion in two steps:determining optimal positions of the fore- vector is normalized with respect to image area.C ground subjects and inserting and removing a certain num- OHrHo,HrOHo is still a constant for ensuring computa- ber of seams. tional stability.We set C to 0.0001 in our experiments. Determination of optimal positions.Given the original image lo,the closest power point and vertical line used Integrating the edge similarity into SSIM,image similar- in rule of thirds to each salient object are first computed. ity between Ir and lo is finally calculated by Since the aesthetic term Ee has an analytical expression,we can easily determine the target location each subject should Es(Ir,Io)=1(Ir,Io).c(Ir,Io) s(Ir,lo)+e(Ir,Io) 2 move towards by maximizing it. (10) Insertion and removal of seams.With the optimal loca- ©2012 The Author(s)Y. W. Guo et al. / Improving Photo Composition Elegantly: Considering Image Similarity During Composition Optimization uniformly set to 1 [WBSS04]. Let µIr and µIo denote mean intensity of Ir and Io separately. σIr and σIo are standard deviations. σIrIo represents the covariance of image vectors of Ir and Io. l(,), c(,), and s(,) are expressed as, l(Ir,Io) = 2µIrµIo +C1 µ2 Ir +µ2 Io +C1 , (6) c(Ir,Io) = 2σIrσIo +C2 σ2 Ir +σ2 Io +C2 , (7) s(Ir,Io) = σIrIo +C3 σIrσIo +C3 , (8) where C1, C2, and C3 are constants for avoiding computational instability. As suggested in [WBSS04], C1 and C2 are set to (K1L) 2 and (K2L) 2 in which L is set to 255 for 8-bit grayscale images, and K1 and K2 can be set as 0.01 and 0.03 separately in implementation. C3 is usually set to C2/2 in practice. The essence of SSIM in contrast to tradition metrics is to compare the structures of two images directly. Studies of cognitive psychology show that human visual perception is very sensitive to the strong edges in natural images. Since retargeting techniques such as seam carving [AS07] or meshguided warping [WTSL08, GLS∗09] may easily destroy or deform important edges, it is vital to reinforce edge similarity in the similarity measure. To account for this, we adopt a measure of edge similarity that compares strong edges and use it as a compliment to structure comparison in SSIM. Sobel operators with a horizontal edge mask and a vertical one are first applied to the given image. This yields two edge maps by exploiting which we can easily compute a gradient magnitude and an orientation for each edge pixel. An edge orientation histogram with 8 bins in 0o − 180o can thus be built, and we use it to compute edge similarity as follows, e(Ir,Io) = σHrHo +C 3 σHrσHo +C 3 , (9) where σHr and σHo denote standard deviations of the histogram vectors of Ir and Io, separately. σHrHo is the covariance of two histograms. Note that, the histogram vector is normalized with respect to image area. C 3 σHrHo,σHrσHo is still a constant for ensuring computational stability. We set C 3 to 0.0001 in our experiments. Integrating the edge similarity into SSIM, image similarity between Ir and Io is finally calculated by Es(Ir,Io) = l(Ir,Io)· c(Ir,Io)· s(Ir,Io) +e(Ir,Io) 2 . (10) 4. Optimization The resulting image is the image maximizing the objective function E(I), Ir = argmaxI E (I). (11) Previous techniques on photo composition enhancement typically transform the images by image retargeting coupled with the cropping operation. We take seam carving [AS07] as the basic operation for improving the composition aesthetics. In addition, we assume here two constraints for meeting users’ requirements in practice. First, image sizes and aspect ratios should not be altered. Cropping is not allowed in this sense as cropping can lead to the loss of valuable image information, even though it can prescribe a straightforward solution to optimal re-composition [BSS10]. Second, salient regions should be free from zooming in or out since zooming in directly inevitably blurs the subject, while zooming out reduces resolution. With the above constraints, we use a heuristic algorithm to find the solution corresponding to the optimal image. Seam carving changes image size and aspect ratio by carving out a series of less noticeable seams. A seam is an optimal path of pixels from top to bottom, or left to right defined in terms of local energy. Such a seam can be found using dynamic programming. Note that, in our application to preserve completely the salient objects, local energy defined on salient regions is valued with the maximal value in the energy function of seam carving. We observe that by iteratively carving out seams on one side of a salient object and inserting the same number of seams on the other side simultaneously, image composition can be modified without changing image size. The key problem is thus to determine k, the number of seams to be removed, together with the k seams to be removed and k new seams to be inserted such that the resulting image I maximizes E(I). We denote {s d i }R i=1 and {s a j}R j=1 as the seams as the candidates for removal and insertion. Each s d i or s a j is labelled as 1 if it is selected, and 0 not selected. The optimization is then formulated as a labelling problem which can be solved by 0-1 mixed integer programming. Global optimization on the parameter space is still computationally expensive and may get stuck in local optima easily. We hereby develop an efficient heuristic algorithm which finds the solution in two steps: determining optimal positions of the foreground subjects and inserting and removing a certain number of seams. Determination of optimal positions. Given the original image Io, the closest power point and vertical line used in rule of thirds to each salient object are first computed. Since the aesthetic term Ee has an analytical expression, we can easily determine the target location each subject should move towards by maximizing it. Insertion and removal of seams. With the optimal loca- c 2012 The Author(s) c 2012 The Eurographics Association and Blackwell Publishing Ltd.