Y.Guo et aL.Computers Graphics 38(2014)174-182 175 A spidery mesh is employed to obtain a simple scene model from the central perspective image using a graphical interface. The animators utilize this incomplete scene information to make animation from the input pictures.Instead of attempting to recover precise geometry,a rough 3D environment is constructed from a single image by applying a statistical framework [8].The model is constructed directly from the learned geometric labels: ground,vertical and sky,on the image.None of the above methods aim to re-generate a new image with high visual quality as if it is captured from a novel viewpoint.In contrast,we only need to partially recover a cuboid dominated 3D representation of the image with moderate user interaction,the whole image is re-rendered by making the rest image region deform in accor- dance with the re-projection of the cuboid structure. Recently.the advances of shape deformation [9,10]and retar- geting techiques [11-15]make it possible to manipulate perspec- Fig.1.Images with cuboid structures. tive by the means of image deformation [1.A 2D image warp is computed by optimizing an energy function such that the entire warp is as shape-preserving as possible,and meanwhile satisfies optimization,we are able to simulate novel viewpoints and to the constraints originated from projective geometry.The user first render the new images with high visual quality by letting the rest annotates an image by marking a number of image space con- image region deform in accordance with the transformation of the straints,with pixel accuracy.User assistance is required to accu- cuboid structure. rately mark the image and manipulate its perspective with a Although high-quality 3D reconstruction from a single image number of image space constraints.Overall eight different types remains difficult,we can recover an approximation of the three- of constraints which may oppose directly each other are incorpo- dimensional cuboid structure easily,with only a few user-specified rated into the energy function.Taking care of these constraints auxiliary lines on the image.It should be noted that although cautiously for efficient optimization poses a challenge for amateur simplistic model of geometry can be recovered by leveraging small users.The problem of manipulating perspective in stereoscopic amounts of annotation [4]or by user annotation assisted scene pairs is addressed in [2].Given a new perspective,correspondence analysis [5,6]for the applications of augmented reality.there is no constraints between stereoscopic image pairs are determined,and doubt that fully automatic recovering the geometry without any user a warp for each image which preserves salient image features and intervention is still challenging.especially for the images with our so guarantees proper stereopsis relative to the new camera is called non-standard cuboid structures where some cuboid edges computed. never exist.We thus allow the users to specify some auxiliary lines Perspective projections are limited to fairly narrow view angles. with trivial user efforts.More importantly,we show that the re- Correction of image deformations incurred by projecting wide projection of this approximated cuboid structure is sufficient to meet fields of view onto a flat 2D display surface is address in[16,171. the requirement of accuracy of viewpoint changes.Given a new Our work is also inspired by the recent efforts on photo viewpoint,the image is rendered by optimizing a quadratic image composition assessment and enhancement [18-20].Most methods warping energy.The energy function incorporates the hard constraint build their measures of visual aesthetics on the rule of thirds of cuboid transformation,and constraints on shape and straight lines which means that an image should be imaged as divided into nine Without significant manual effort,the newly perspective image equal parts by two equally spaced horizontal lines and two vertical rendered is nearly geometrically accurate,and visually pleasing. lines,and important compositional elements should be placed Applications:Firstly,our technique can be used for correcting along these lines or their intersections.Bhattacharya et al.[18 those slanted structures of photos taken by casual photographers. learn a support vector regression model for capturing aesthetics. Secondly,unlike previous image deformation driven methods,we Image quality is improved by recomposing the salient object onto can generate novel images under key viewpoints around the the inpainted background or by using a visual weight balancing viewpoint of input image,with given viewing angles.This enables technique.Liu et al.[20]modify image composition by using a us to design an interface through which the user can watch the compound operator of crop-and-retarget and seek the solution by scene by changing viewpoints smoothly on a viewing sphere, particle swarm optimization. mimicking 3D browsing experience.The images under key view- points are interpolated to produce intermediate results. To summarize,our main contributions are as follows: 3.View manipulation of the cuboid structure .Present an algorithm for manipulating views of cuboid- structured images with very little user effort. Our view manipulation method is specifically designed to Show that re-projection of the approximated cuboid structure optimize viewpoints of those images that show cuboid-dominated is sufficient to meet the requirement of viewpoint change. three-dimensional structures.Extracting a 3D representation from a Provide an interface that allows the users to watch the scene single-view image depicting a 3D object has been a longstanding under new viewpoints on a viewing sphere interactively. goal of computer vision.It has been shown recently that 3D cuboids in single-view images can be automatically localized by using a discriminative parts-based detector [21.We allow the users to interactively specify projected lines of the latent cuboid structure on 2.Related work the image,with which we estimate an approximation of the cuboid geometry.Hough transform and Canny edge detector are used Manipulation of the perspective in a photograph for the tasks to assist users and to reduce interaction errors in this process of touring into the pictures is made possible by Horry et al.[7]. We show that,given a new viewpoint,the re-projection of thisoptimization, we are able to simulate novel viewpoints and to render the new images with high visual quality by letting the rest image region deform in accordance with the transformation of the cuboid structure. Although high-quality 3D reconstruction from a single image remains difficult, we can recover an approximation of the three￾dimensional cuboid structure easily, with only a few user-specified auxiliary lines on the image. It should be noted that although simplistic model of geometry can be recovered by leveraging small amounts of annotation [4] or by user annotation assisted scene analysis [5,6] for the applications of augmented reality, there is no doubt that fully automatic recovering the geometry without any user intervention is still challenging, especially for the images with our so called non-standard cuboid structures where some cuboid edges never exist. We thus allow the users to specify some auxiliary lines with trivial user efforts. More importantly, we show that the re￾projection of this approximated cuboid structure is sufficient to meet the requirement of accuracy of viewpoint changes. Given a new viewpoint, the image is rendered by optimizing a quadratic image warping energy. The energy function incorporates the hard constraint of cuboid transformation, and constraints on shape and straight lines. Without significant manual effort, the newly perspective image rendered is nearly geometrically accurate, and visually pleasing. Applications: Firstly, our technique can be used for correcting those slanted structures of photos taken by casual photographers. Secondly, unlike previous image deformation driven methods, we can generate novel images under key viewpoints around the viewpoint of input image, with given viewing angles. This enables us to design an interface through which the user can watch the scene by changing viewpoints smoothly on a viewing sphere, mimicking 3D browsing experience. The images under key view￾points are interpolated to produce intermediate results. To summarize, our main contributions are as follows: Present an algorithm for manipulating views of cuboid￾structured images with very little user effort. Show that re-projection of the approximated cuboid structure is sufficient to meet the requirement of viewpoint change. Provide an interface that allows the users to watch the scene under new viewpoints on a viewing sphere interactively. 2. Related work Manipulation of the perspective in a photograph for the tasks of touring into the pictures is made possible by Horry et al. [7]. A spidery mesh is employed to obtain a simple scene model from the central perspective image using a graphical interface. The animators utilize this incomplete scene information to make animation from the input pictures. Instead of attempting to recover precise geometry, a rough 3D environment is constructed from a single image by applying a statistical framework [8]. The model is constructed directly from the learned geometric labels: ground, vertical and sky, on the image. None of the above methods aim to re-generate a new image with high visual quality as if it is captured from a novel viewpoint. In contrast, we only need to partially recover a cuboid dominated 3D representation of the image with moderate user interaction, the whole image is re-rendered by making the rest image region deform in accor￾dance with the re-projection of the cuboid structure. Recently, the advances of shape deformation [9,10] and retar￾geting techiques [11–15] make it possible to manipulate perspec￾tive by the means of image deformation [1]. A 2D image warp is computed by optimizing an energy function such that the entire warp is as shape-preserving as possible, and meanwhile satisfies the constraints originated from projective geometry. The user first annotates an image by marking a number of image space con￾straints, with pixel accuracy. User assistance is required to accu￾rately mark the image and manipulate its perspective with a number of image space constraints. Overall eight different types of constraints which may oppose directly each other are incorpo￾rated into the energy function. Taking care of these constraints cautiously for efficient optimization poses a challenge for amateur users. The problem of manipulating perspective in stereoscopic pairs is addressed in [2]. Given a new perspective, correspondence constraints between stereoscopic image pairs are determined, and a warp for each image which preserves salient image features and guarantees proper stereopsis relative to the new camera is computed. Perspective projections are limited to fairly narrow view angles. Correction of image deformations incurred by projecting wide fields of view onto a flat 2D display surface is address in [16,17]. Our work is also inspired by the recent efforts on photo composition assessment and enhancement [18–20]. Most methods build their measures of visual aesthetics on the rule of thirds which means that an image should be imaged as divided into nine equal parts by two equally spaced horizontal lines and two vertical lines, and important compositional elements should be placed along these lines or their intersections. Bhattacharya et al. [18] learn a support vector regression model for capturing aesthetics. Image quality is improved by recomposing the salient object onto the inpainted background or by using a visual weight balancing technique. Liu et al. [20] modify image composition by using a compound operator of crop-and-retarget and seek the solution by particle swarm optimization. 3. View manipulation of the cuboid structure Our view manipulation method is specifically designed to optimize viewpoints of those images that show cuboid-dominated three-dimensional structures. Extracting a 3D representation from a single-view image depicting a 3D object has been a longstanding goal of computer vision. It has been shown recently that 3D cuboids in single-view images can be automatically localized by using a discriminative parts-based detector [21]. We allow the users to interactively specify projected lines of the latent cuboid structure on the image, with which we estimate an approximation of the cuboid geometry. Hough transform and Canny edge detector are used to assist users and to reduce interaction errors in this process. We show that, given a new viewpoint, the re-projection of this Fig. 1. Images with cuboid structures. Y. Guo et al. / Computers & Graphics 38 (2014) 174–182 175