Computers Graphics 38 (2014)174-182 Contents lists available at ScienceDirect Computers Graphics ELSEVIER journal homepage:www.elsevier.com/locate/cag CAD/Graphics 2013 Efficient view manipulation for cuboid-structured images CrossMark Yanwen Guo .*Guiping Zhang,Zili Lan3,Wenping Wang State Key Lab for Novel Software Technology.Nanjing University.PR China Department of Computer Science,The University of Hong Kong Hong Kong ARTICLE INFO ABSTRACT Article history: We present in this paper an efficient algorithm for manipulating the viewpoints of cuboid-structured Received 31 July 2013 images with moderate user interaction.Such images are very popular,and we first recover an Received in revised form approximate geometric model with the prior knowledge of the latent cuboid.While this approximated 27 October 2013 Accepted 28 October 2013 cuboid structure does not provide an accurate scene reconstruction,we demonstrate that it is sufficient Available online 13 November 2013 to re-render the images realistically under new viewpoints in a nearly geometrically accurate manner. The new image with high visual quality is generated by making the rest image region deform in Keywords: accordance with the re-projected cuboid structure,via a triangular mesh deformation scheme.The View manipulation energy function has been carefully designed to be a quadratic function so that it can be efficiently Mesh deformation Optimization minimized via solving a sparse linear system.We verify the effectiveness of our technique through testing images with standard and non-standard cuboid structures,and demonstrate an application of upright adjustment of photographs and a user interface which enables the user to watch the scene under new viewpoints on a viewing sphere interactively. 2013 Elsevier Ltd.All rights reserved 1.Introduction Stereoscopic devices and content relying on stereopsis are now widely available,and the problem of manipulating perspective in Advances in imaging technology and hardware improvement of stereoscopic pairs is addressed in [2].Assuming that depth digital cameras result in continuous improvements of image quality. variations of the scene relative to its distance from the camera People can take high quality photos at high resolutions without are small,slanted man-made structures can be straightened up by always suffering from noises,low contrast,and blur that may an improved homography model [3]. degrade photo quality,more easily than before.However,photos We do not intend to study the aesthetics of whether or not a taken by amateur photographers are often with bad viewpoints.for photograph looks visually pleasing under the current viewpoint. instance slanted man-made structures and unbalanced compositions, Instead,our goal is to enable the generation of novel images with making scenes look dull and less vivid.On the other hand,when new viewpoints given only a single image as input,with moderate looking at a photo shared by friends or downloaded from Flicker or user assistance.To this end,our primary observation is that many Photobucket,people may imagine naturally what the scene looks like images of man-made scenes exhibit the cuboid dominated three- if it is taken under a new viewpoint.Automatic optimization of the dimensional structures,in which projections of two perpendicular viewpoint of a given photograph is thus desired. planes dominating the latent three-dimensional geometry.occu- For rendering the image with a novel viewpoint,direct repro- pying the major part of an image.A pair of projected parallel lines jection of the 3D latent scene remains elusive since accurate in each plane can be found in the image.Such an image either reconstruction of the whole scene is still challenging.Recent itself has a cuboid structure or its scene is dominated by a cuboid- efforts have been made to optimize perspective or to imitate re- like object.As shown in Fig.1,the cuboid-structured images are projection by means of image transformation.Manipulation of very popular,for example,the photos of buildings(upper left and photographic perspective is enabled in [1]by combining recent lower left).indoor scenes (upper right),apartments,and buses. image warping techniques and constraints from projective geo- Essentially,some photos exhibit latent cuboid structures.For metry.Heavy user assistance based on understanding of the basic example,the lower right photo of Fig.1 is such an image since principles of perspective construction is often required to accu- we can easily construct two perpendicular planes by specifying rately mark the image with a number of image space constraints. auxiliary lines shown as the dotted lines in this photo,even though physically one of the two planes containing the auxiliary lines does not exist.Such a cuboid structure is the major visual cue .Corresponding author.Tel.:+86 1391 3028 596:fax:+86 25 896 86596 to depict a three-dimensional scene and to convey perspective.By E-mail addresses:ywguo.nju@gmail.com,ywguo@nju.edu.cn (Y.Guo). manipulating the cuboid structure reconstructed by acceptable 0097-8493/S-see front matter e 2013 Elsevier Ltd.All rights reserved. htp:/dx.doi.org10.10160.cag2013.10.038CAD/Graphics 2013 Efficient view manipulation for cuboid-structured images Yanwen Guo a,n , Guiping Zhang a , Zili Lan a , Wenping Wang b a State Key Lab for Novel Software Technology, Nanjing University, PR China b Department of Computer Science, The University of Hong Kong, Hong Kong article info Article history: Received 31 July 2013 Received in revised form 27 October 2013 Accepted 28 October 2013 Available online 13 November 2013 Keywords: View manipulation Mesh deformation Optimization abstract We present in this paper an efficient algorithm for manipulating the viewpoints of cuboid-structured images with moderate user interaction. Such images are very popular, and we first recover an approximate geometric model with the prior knowledge of the latent cuboid. While this approximated cuboid structure does not provide an accurate scene reconstruction, we demonstrate that it is sufficient to re-render the images realistically under new viewpoints in a nearly geometrically accurate manner. The new image with high visual quality is generated by making the rest image region deform in accordance with the re-projected cuboid structure, via a triangular mesh deformation scheme. The energy function has been carefully designed to be a quadratic function so that it can be efficiently minimized via solving a sparse linear system. We verify the effectiveness of our technique through testing images with standard and non-standard cuboid structures, and demonstrate an application of upright adjustment of photographs and a user interface which enables the user to watch the scene under new viewpoints on a viewing sphere interactively. & 2013 Elsevier Ltd. All rights reserved. 1. Introduction Advances in imaging technology and hardware improvement of digital cameras result in continuous improvements of image quality. People can take high quality photos at high resolutions without always suffering from noises, low contrast, and blur that may degrade photo quality, more easily than before. However, photos taken by amateur photographers are often with bad viewpoints, for instance slanted man-made structures and unbalanced compositions, making scenes look dull and less vivid. On the other hand, when looking at a photo shared by friends or downloaded from Flicker or Photobucket, people may imagine naturally what the scene looks like if it is taken under a new viewpoint. Automatic optimization of the viewpoint of a given photograph is thus desired. For rendering the image with a novel viewpoint, direct repro￾jection of the 3D latent scene remains elusive since accurate reconstruction of the whole scene is still challenging. Recent efforts have been made to optimize perspective or to imitate re￾projection by means of image transformation. Manipulation of photographic perspective is enabled in [1] by combining recent image warping techniques and constraints from projective geo￾metry. Heavy user assistance based on understanding of the basic principles of perspective construction is often required to accu￾rately mark the image with a number of image space constraints. Stereoscopic devices and content relying on stereopsis are now widely available, and the problem of manipulating perspective in stereoscopic pairs is addressed in [2]. Assuming that depth variations of the scene relative to its distance from the camera are small, slanted man-made structures can be straightened up by an improved homography model [3]. We do not intend to study the aesthetics of whether or not a photograph looks visually pleasing under the current viewpoint. Instead, our goal is to enable the generation of novel images with new viewpoints given only a single image as input, with moderate user assistance. To this end, our primary observation is that many images of man-made scenes exhibit the cuboid dominated three￾dimensional structures, in which projections of two perpendicular planes dominating the latent three-dimensional geometry, occu￾pying the major part of an image. A pair of projected parallel lines in each plane can be found in the image. Such an image either itself has a cuboid structure or its scene is dominated by a cuboid￾like object. As shown in Fig. 1, the cuboid-structured images are very popular, for example, the photos of buildings (upper left and lower left), indoor scenes (upper right), apartments, and buses. Essentially, some photos exhibit latent cuboid structures. For example, the lower right photo of Fig. 1 is such an image since we can easily construct two perpendicular planes by specifying auxiliary lines shown as the dotted lines in this photo, even though physically one of the two planes containing the auxiliary lines does not exist. Such a cuboid structure is the major visual cue to depict a three-dimensional scene and to convey perspective. By manipulating the cuboid structure reconstructed by acceptable Contents lists available at ScienceDirect journal homepage: www.elsevier.com/locate/cag Computers & Graphics 0097-8493/$ - see front matter & 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.cag.2013.10.038 n Corresponding author. Tel.: þ86 1391 3028 596; fax: þ86 25 896 86596. E-mail addresses: ywguo.nju@gmail.com, ywguo@nju.edu.cn (Y. Guo). Computers & Graphics 38 (2014) 174–182