176 Y.Guo et al.Computers Graphics 38(2014)174-182 system.The new imaging plane passing through the world center o is I'whose normal vector is 0o'.Let (Qo....Qs)denote the real coordinates of the 3D cuboid,corresponding to the projected points (po.....Ps)on the input image.We prove that the 2D projections of (Po.....Ps)on the new imaging plane I'is identical to the projections of (Qo.....Qs).by eliminating translation.To achieve this,we only need to prove Fig.2.Left:a standard case of the projection of a cuboid structure.Note that paP1=9091: 2) IPop4l=IpPsl and lPopal=IpiPal are not required.paps and pap3 are not necessa- rily the projections of two vertical edges of the cuboid.Right:the 3D cuboid structure. where PoP;represents the projected vector of PoP on I',and qoq denotes the projected vector of QoQ on lo the imaging plane with approximated cuboid structure is sufficient to meet the require- focal length f if (Qo.Qs)are given (see Fig.3). ment of accuracy of viewpoint change. We should rotate the camera around 0o(0,0.z)if (Qo.....Qs} are known.In this case,the new viewpoint is 0o.We have 3.1.A standard case IOoFI=1000ol.Let lo passing through Oo be an auxiliary plane 3.1.1.Cuboid reconstruction whose normal is 0o0o.Let PooPo be the projected vector of For ease of exposition,it is initially assumed that we can find, QoQ on Io.we get on the image plane,the projections of a vertical edge and two pairs of parallel edges with two interaction points on the vertical edge of a standard cuboid structure,as shown in Fig.2.Without 9o=OaFPaoPa (3) loss of generality,we assume that the imaging plane is placed It is obvious that (0o'pi)and (OoOoPo)are similar triangles. along with the XY plane of the world coordinate system with its center at the world origin.The camera is stationed at Z-axis with A(poopi)and A(PoooPo)are similar triangles.We get center of projection F(0.0.f).fis the focal length of the camera.Let [Po....,Ps}and (po.....ps}denote six key vertexes on the latent PoP O'P 00 f (4) cuboid facing the camera and their projections on the image, PQoPQ1 OoPQ OQOo =0o万 respectively.(Po.....Ps)are specified by user on the image.Po and P are two corners of the cuboid shared by two perpendicular Combining(3)and (4).we get pop=gog1. planes P1(P1,Po,P4,Ps)and P2(Po,P1,P2,P3)on the three-dimen- sional cuboid.Since PoP3//P1P2 in 3D space,the extensions of Pop3 3.2.A more complex case and pipz will meet at a vanishing point c1.except for the special case PoP3//piP2.Similarly,the extensions of piPs and pop meet at In some images we cannot find,on the projected vertical edge, another one c2.Imagine that PoP3 and P1P2 meet at a point at two interaction points of the projected parallel edges.Fig.4 shows infinity whose projection on the imaging plane is c.The extension such an example.To tackle this issue,we first compute the of Fc will meet with PoP3 and P P2 at this point as well.We thus vanishing points of the projected parallel line segments.Two have FC1//PoP3.FC1//P1P2.and Fc1//P1.Similarly,Fc2 is parallel to auxiliary lines,shown as dotted segments in Fig.4,which pass P2,and Fc is perpendicular to Fc2 with which fcan be obtained easily. through the corresponding vanishing points can be drawn.This It is still impossible to recover the accurate geometry and case is then converted to the standard case we have described position of the 3D cuboid structure without any other prior previously.We then compute equations of the two perpendicular knowledge.By making reasonable assumptions,we wish to gen- erate an approximation of the structure which is exactly the same as the accurate one without considering the scale difference.Given any new viewpoint,we show that the re-projection of this approximated cuboid structure is sufficient to meet the require- ment of accuracy of viewpoint changes. Considering that IFpol is proportional to IFPol,coordinate of Po 00 can be obtained by setting the ratio of IFpol to IFPol to a constant. We set it to 1 in our experiments.P can be represented by parametric coordinate with F and p.We then compute (P1.....Ps) 00 by exploiting the geometric relationships PoP1 L (PoP3,PoP4,P1P2,P1Ps),PoP4//PsP1,PoP3//P1P2.(1) 3.1.2.Analysis of accuracy Given a new viewpoint,we assume that focal length f of the camera remains fixed,since zooming in and out can be easily imitated by upsampling and downsampling the image.Without loss of generality,center of the scene is placed at(0,0,zP)with Zpo the z-coordinate of Po.Recall that we set the ratio of IFpol to IFPol to 1.Therefore,center of the scene is (0,0,0)which is in accordance with the world center O.Let us denote the new viewpoint by o'(f sin o cos 6.f sin o sine.f cos o)where (0.) Fig.3.Projection of PoPi on the new imaging plane I'is identical to the projected is the polar angle and azimuthal angle in spherical coordinate QoQ on lo by eliminating translation.approximated cuboid structure is sufficient to meet the require￾ment of accuracy of viewpoint change. 3.1. A standard case 3.1.1. Cuboid reconstruction For ease of exposition, it is initially assumed that we can find, on the image plane, the projections of a vertical edge and two pairs of parallel edges with two interaction points on the vertical edge of a standard cuboid structure, as shown in Fig. 2. Without loss of generality, we assume that the imaging plane is placed along with the XY plane of the world coordinate system with its center at the world origin. The camera is stationed at Z-axis with center of projection Fð0; 0; fÞ. f is the focal length of the camera. Let fP0; …; P5g and fp0; …; p5g denote six key vertexes on the latent cuboid facing the camera and their projections on the image, respectively. fp0; …; p5g are specified by user on the image. P0 and P1 are two corners of the cuboid shared by two perpendicular planes P1ðP1; P0; P4; P5Þ and P2ðP0; P1; P2; P3Þ on the three-dimen￾sional cuboid. Since P0P3==P1P2 in 3D space, the extensions of p0p3 and p1p2 will meet at a vanishing point c1, except for the special case p0p3==p1p2. Similarly, the extensions of p1p5 and p0p4 meet at another one c2. Imagine that P0P3 and P1P2 meet at a point at infinity whose projection on the imaging plane is c1. The extension of Fc1 will meet with P0P3 and P1P2 at this point as well. We thus have Fc1==P0P3, Fc1==P1P2, and Fc1==P1. Similarly, Fc2 is parallel to P2, and Fc1 is perpendicular to Fc2 with which f can be obtained easily. It is still impossible to recover the accurate geometry and position of the 3D cuboid structure without any other prior knowledge. By making reasonable assumptions, we wish to gen￾erate an approximation of the structure which is exactly the same as the accurate one without considering the scale difference. Given any new viewpoint, we show that the re-projection of this approximated cuboid structure is sufficient to meet the require￾ment of accuracy of viewpoint changes. Considering that jFp0j is proportional to jFP0j, coordinate of P0 can be obtained by setting the ratio of jFp0j to jFP0j to a constant. We set it to 1 in our experiments. Pi can be represented by parametric coordinate with F and pi. We then compute fP1; …; P5g by exploiting the geometric relationships P0P1 ? fP0P3; P0P4; P1P2; P1P5g; P0P4==P5P1; P0P3==P1P2: ð1Þ 3.1.2. Analysis of accuracy Given a new viewpoint, we assume that focal length f of the camera remains fixed, since zooming in and out can be easily imitated by upsampling and downsampling the image. Without loss of generality, center of the scene is placed at ð0; 0; zP0 Þ with zP0 the z-coordinate of P0. Recall that we set the ratio of jFp0j to jFP0j to 1. Therefore, center of the scene is ð0; 0; 0Þ which is in accordance with the world center O. Let us denote the new viewpoint by O′ðf sin φ cos θ; f sin φ sin θ; f cos φÞ where ðθ; φÞ is the polar angle and azimuthal angle in spherical coordinate system. The new imaging plane passing through the world center O is I′ whose normal vector is OO′

!. Let fQ0; …;Q5g denote the real coordinates of the 3D cuboid, corresponding to the projected points fp0; …; p5g on the input image. We prove that the 2D projections of fP0; …; P5g on the new imaging plane I′ is identical to the projections of fQ0; …;Q5g, by eliminating translation. To achieve this, we only need to prove p′ 0p′ 1

! ¼ q′ 0q′ 1

!; ð2Þ where p′ 0p′ 1

! represents the projected vector of P0P1

! on I′, and q′ 0q′ 1

! denotes the projected vector of Q0Q1


! on I ′ Q the imaging plane with focal length f if fQ0; …;Q5g are given (see Fig. 3). We should rotate the camera around OQ ð0; 0; zQ0 Þ if fQ0; …;Q5g are known. In this case, the new viewpoint is O′ Q . We have jOQ Fj¼jOQO′ Q j. Let I ′ Q passing through OQ be an auxiliary plane whose normal is OQO′ Q


!. Let p′ Q0p′ Q1



! be the projected vector of Q0Q1


! on I ′ Q , we get q′ 0q′ 1

! ¼ f jOQ Fj p′ Q0p′ Q1



!: ð3Þ It is obvious that▵ðOO′p′ 1Þ and▵ðOQO′ Q p′ Q1Þ are similar triangles. ▵ðp′ 0O′p′ 1Þ and▵ðp′ Q0O′ Q p′ Q1Þ are similar triangles. We get p′ 0p′ 1

! p′ Q0p′ Q1



! ¼ O′p′ 1

! O′ Q p′ Q1



! ¼ OO′

! OQO′ Q


! ¼ f jOQ Fj : ð4Þ Combining (3) and (4), we get p′ 0p′ 1

! ¼ q′ 0q′ 1

!. 3.2. A more complex case In some images we cannot find, on the projected vertical edge, two interaction points of the projected parallel edges. Fig. 4 shows such an example. To tackle this issue, we first compute the vanishing points of the projected parallel line segments. Two auxiliary lines, shown as dotted segments in Fig. 4, which pass through the corresponding vanishing points can be drawn. This case is then converted to the standard case we have described previously. We then compute equations of the two perpendicular p0 1p 2p 3p 4 p 5 p c2 c1 P0 1 2P 3 P P4 P5 P Fig. 2. Left: a standard case of the projection of a cuboid structure. Note that, jp0p4j¼jp1p5j and jp0p3j¼jp1p2j are not required. p4p5 and p2p3 are not necessa￾rily the projections of two vertical edges of the cuboid. Right: the 3D cuboid structure. F Q0 Q1 P1 P0 O ' 0 p ' 1p ' 0q ' 1q OQ ' OQ ' Q0 p ' Q1 p X Y Z I O' ' I' Q I' q Fig. 3. Projection of P0P1 on the new imaging plane I′ is identical to the projected Q0Q1 on I ′ Q by eliminating translation. 176 Y. Guo et al. / Computers & Graphics 38 (2014) 174–182