mage Processing and Computer Vision Chapter 10: Pose estimation by the iterative method Pose estimation vo.a
Image Processing and Computer Vision Chapter 10: Pose estimation by the iterative method Pose estimation V0.a 1
Overview Define the terms Define structure from motion sem Methods for sem Define pose estimation and why we need to study it Newton 's method Iterative algorithm for pose estimation Pose estimation vo.a
Overview • Define the terms • Define Structure From Motion SFM • Methods for SFM • Define pose estimation, and why we need to study it • Newton's method • Iterative algorithm for pose estimation Pose estimation V0.a 2
Intro) I Motivation I Pose est. I Newton's method I Iterative method Define the terms 3 D Model=X1=ⅨXyZ7 where i=feature index X=2=[92,126209] 1 2.N features X can be found by manual Xi=1=[102, 18,23 measurement Pose 6t is the Rotation(R) and Translation(t)of the object at a time t, where u= horizontal image position 3x3/3x1t v=vertical image position gti= lu,v] is the image point of the ith 3D feature at time t Pose estimation vo.a
Intro. | Motivation | Pose est.| Newton’s method | Iterative method Define the terms 3 • 3D Model=Xi =[X,Y,Z]i T : where i=feature index =1,2…N features. • X can be found by manual measurement • Pose t is the Rotation (R) and Translation (T) of the object at a time t, where t={R3x3,T3x1} t • q t i = [u,v] t i is the image point of the i th 3D feature at time t Xi=1=[102,18,23]T X Y Z Xi=2=[92,126,209]T u= horizontal image position, v=vertical image position Pose estimation V0.a
Intro) Motivation I Pose est. I Newton's method I Iterative method What is Structure from motion sem? 3D Model=X;: where i=feature index =1, 2.N features t=3 Time(t) 41=N t=T i=N t=1 2 R 3×313×1t=2 (R3 (R3 3×313×1)t=3 (R33,I,x1 3×313×1t=1 Capture images at time index t=1, 2, Extract image features qi, i=feature index =12, 3.N To Find: Pose at timet is(R3x37I3xI) and structure of the object(X,Y, Z of each 3-D feature on the object) 4 Pese-estimatien Oa
Intro. | Motivation | Pose est.| Newton’s method | Iterative method What is Structure From Motion SFM? • 3D Model=Xi : where i=feature index =1,2…N features 4 t=1 t= 2 t=3 … Time (t) 3 3 3 1 1 ( , ) R T t= structure of the object (X,Y,Z of each 3- D feature on the object) To Find :Pose at time is , and Extract image features feature index 12 3 Capture images at time index 1 2 3 3 3 1 t t i t (R T ) q ,i , ..N t , ,...Γ = = = 1 2 = = t i q 1 1 = = t i q =1 = t i N q = = t qi N 3 3 3 1 2 ( , ) R T t= 3 3 3 1 3 ( , ) R T t= R T t= ( , ) 3 3 3 1 Pose estimation V0.a
Intro)I Motivation I Pose est. I Newton's method I Iterative method Methods of structure from motion sem: 3D reconstruction from n-frames Factorization(linear, fast, not too accurate) Bundle adjustment ba (slower but more accurate) can use factorization results as the first guess Non-linear iterative methods are more accurate than linear method require first guess(e.g. From factorization) Many different implementations but the concept is the same Two-step Bundle Adjustment(a special form of Bundle adjustment BA) Iterative pose estimation Iterative structure reconstruction Pose estimation vo.a 5
Intro. | Motivation | Pose est.| Newton’s method | Iterative method Methods of Structure From Motion SFM :3D reconstruction from N-frames • Factorization (linear, fast, not too accurate) • Bundle adjustment BA (slower but more accurate), can use factorization results as the first guess. – Non-linear iterative methods are more accurate than linear method, require first guess (e.g. From factorization). – Many different implementations, but the concept is the same. – Two-step Bundle Adjustment (a special form of Bundle adjustment BA) • Iterative pose estimation • Iterative structure reconstruction Pose estimation V0.a 5
Intro.MotivationPoseest.Newton's method| Iterative method Motivation of studying pose estimation In order to understand bundle adjustment for 3-d model structure reconstruction from -frames, we need to understand pose estimation first Pose estimation problem definition there are n features in a known 3D object For pose estimation We take / pictures of the object at different views.(Assume object stationary, camera moving at time t=12…,F Input We know there are n 3-d model points of the object: X,Y, Z)12. N and Image sequence I,I2,Ir: Each image has N 2-D image feature points Output (pose=rotation and translation the camera) 0,1=(R Tt1, 2=(R=,.,0 =[ R, Tt=r poses are found Pose estimation vo.a 6
Intro. | Motivation | Pose est.| Newton’s method | Iterative method Motivation of studying pose estimation • In order to understand bundle adjustment for 3-D model structure reconstruction from -frames, we need to understand pose estimation first. • Pose estimation problem definition: There are N features in a known 3D object . • For pose estimation: We take pictures of the object at different views. (Assume object stationary, camera moving at time t=12,.., . • Input : • We know there are N 3-D model points of the object : (X, Y, Z)1,2,..,N, and • Image sequence I1 ,I2 ,…I. Each image has N 2-D image feature points • Output (pose=rotation and translation the camera) • t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= poses are found Pose estimation V0.a 6
Intro.MotivationPoseest.Newton's method| Iterative method Example: Bundle adjustment 3d reconstruction (see also http://www.cse.cuhk.edu.hk/khwong/demo/index.html) Grand Canyon Demo flask Robot http://www.youtube.com/watch?v=2klfriliojc http://www.youtube.com/watch?v=xgcnv--wf2k http://www.youtube.com/watch?v=onx4cyyyyri http://www.youtube.com/watch?v=4h1pn2dis6g Pose estimation vo.a 7
Intro. | Motivation | Pose est.| Newton’s method | Iterative method Example: Bundle adjustment 3D reconstruction (see also http://www.cse.cuhk.edu.hk/khwong/demo/index.html) • Grand Canyon Demo • Flask • Robot Pose estimation V0.a 7 http://www.youtube.com/watch?v=2KLFRILlOjc http://www.youtube.com/watch?v=4h1pN2DIs6g http://www.youtube.com/watch?v=ONx4cyYYyrI http://www.youtube.com/watch?v=xgCnV--wf2k
Intro. MOtivation) Pose est. I Newton's method I Iterative method The iterative SFM alternating(2-stepbundle adjustment Break down the system into two phases SM1: find pose estimation phase --SFM2: find model estimation phase Initialize first guess of model The first guess is a flat model perpendicular to the image and is Zinit away ( e. g. Zinit =0. 5 meters or any reasonable guess) Iterative while( err is not small SFM1: find pose phase SFM2 find model phase Measurement error(Error(model and pose stabilized Pose estimation vo.a 8
Intro. | Motivation | Pose est.| Newton’s method | Iterative method The iterative SFM alternating (2-step)bundle adjustment • Break down the system into two phases: --SFM1: find pose estimation phase --SFM2: find model estimation phase • Initialize first guess of model – The first guess is a flat model perpendicular to the image and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess) • Iterative while ( Err is not small ) • { – SFM1: find pose phase – SFM2: find model phase – Measurement error(Err) or(model and pose stabilized) • } Pose estimation V0.a 8
Intro.MotivationPoseest.Newton's method| Iterative method Define SFM1: the pose estimation algorithm (assume the model is found or given, or guessed Use Klt (Feature Detector interface( or lkdemo. c)in opencv, or http://www.cs.ubcca/mlowe/keypoints/toobtainfeaturesin[u,v]t There are t=1, 2 ,, I image frames So there are 0 1 (R, Ts1, 0 =2=(R,+2,.,0 =R, T]=r poses Given: focal length f and one model M=XY,Zl with j=1,., Features Initialize first guess of model The first guess is a flat model perpendicular to the image and is Zinit away(e.g. Zinit =0.5 meters or any reasonable guess For(=1; t<=/; t++//runs Itimes here i or every time frame t, use all N features, run SFM1 once; so SFM1SFM1: find pose: to find 8) After the above is run e=1 R, Tt=, 0+2=R,TIts,.,0.=R, Tt r poses are found Pose estimation vo.a
Intro. | Motivation | Pose est.| Newton’s method | Iterative method Define SFM1: the pose estimation algorithm (assume the model is found or given, or guessed) • Use KLT (FeatureDetector interface (or lkdemo.c) in opencv, or http://www.cs.ubc.ca/~lowe/keypoints/) to obtain features in [u,v] T • There are t=1,2,…,image frames, • So there are t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= poses. • Given: focal length f and one model Mi=[X,Y,Z]I ,with i=1,..,N features • Initialize first guess of model – The first guess is a flat model perpendicular to the image and is Zinit away (e.g. Zinit = 0.5 meters or any reasonable guess) – For (t=1; t<=; t++) // runs times here – { (for every time frame t, use all N features, run SFM1 once); – so SFM1 {SFM1: find pose : to find t } – } – After the above is run – t=1={R,T} t=1 , t=2={R,T} t=2 , …., t=={R,T} t= poses are found Pose estimation V0.a 9
Intro. I Motivation I Pose est. I Newton's method I Iterative method Define SFM2 the structure estimation algorithm (assume poses are found or given o be discussed in the next chapter: Bundle adjustment Similar to pose estimation In pose estimation model is known pose is unknown Here(model finding by the iterative method Assume pose is known model is unknown The ideas of the algorithms are similar Pose estimation vo.a
Intro. | Motivation | Pose est.| Newton’s method | Iterative method Define SFM2: the structure estimation algorithm (assume poses are found or given) To be discussed in the next chapter : Bundle adjustment • Similar to pose estimation. – In pose estimation: model is known, pose is unknown. – Here (Model finding by the iterative method) Assume pose is known, model is unknown. – The ideas of the algorithms are similar. Pose estimation V0.a 10