Computer Vision and Image Understanding 118(2014)128-139 Contents lists available at ScienceDirect Computer Vision and Image Understanding ELSEVIER journal homepage:www.elsevier.com/locate/cviu Object tracking using learned feature manifolds* CrossMark Yanwen Guo*,Ye Chen3,Feng Tang,Ang Li,Weitao Luo,Mingming Liu National Key Lab for Novel Software Technology,Nanjing University.Nanjing 210023.PR China P Hewlett-Packard Laboratories,Palo Alto,CA 94304.USA University of Maryland,College Park,MD 20740.USA ARTICLE INFO ABSTRACT Article history: Local feature based object tracking approaches have been promising in solving the tracking problems Received 10 August 2012 such as occlusions and illumination variations.However,existing approaches typically model feature Accepted 30 September 2013 variations using prototypes,and this discrete representation cannot capture the gradual changing prop- Available online 17 October 2013 erty of local appearance.In this paper,we propose to model each local feature as a feature manifold to characterize the smooth changing behavior of the feature descriptor.The manifold is constructed from Keywords: a series of transformed images simulating possible variations of the feature being tracked.We propose Feature manifold to build a collection of linear subspaces which approximate the original manifold as a low dimensional SIFT Tracking representation.This representation is used for object tracking.Object location is located by a feature- to-manifold matching process.Our tracking method can update the manifold status,add new feature manifolds and remove expiring ones adaptively according to object appearance.We show both qualita- tively and quantitatively this representation significantly improves the tracking performance under occlusions and appearance variations using standard tracking dataset. 2013 Elsevier Inc.All rights reserved. 1.Introduction Object dynamics model how the object appearance evolves over time to be able to handle appearance variations.The two problems Object tracking is a central problem in computer vision with are usually coupled together:the object representation should be many applications,such as activity analysis,automated surveil- designed to be easily updated to model appearance variations, lance,traffic monitoring,and human-computer interaction.It is while the object dynamics should be able to take advantage of essentially the problem of finding the most likely estimate of the the characteristics of object representation for model update. object state given a sequence of observations.Object tracking is Traditional methods for representing the object,such as global challenging because of: histogram based approach in meanshift tracking [1]and PCA sub- space based approach in EigenTracking[2],are global approaches Complex object appearance.The object may have complicated which describe the object to be tracked as a whole.Such methods appearance which is hard to model.Furthermore,it may work well in many practical applications,but have several intrinsic undergo significant changes due to the pose and scale variations limitations.First,it is usually very difficult for a global representa- as well as non-rigid object motions. tion to capture local details and as a result unable to model com- Occlusions.The object may be occluded by the background or plex appearances.Second,global representations are not robust other moving objects,making it difficult to be localized. to partial occlusion.Once the object is occluded,the whole feature Complex object motion.This is caused by either the moving pat- vector of object representation is affected.Third,global representa- tern of the object or by camera motion accompanied by object tions are hard to update. motion. Recently,local representations have opened a promising direc- tion to solve these problems by representing an object as a set of There are two key components in an object tracking algorithm: local parts or sparse local features.Part-based trackers generally object representation and dynamics.Object representation tries to use sets of connected or global visual properties incorporated local model the object as accurately as possible so that the tracking parts or components [3-6.The parts used for object representa- algorithm can accurately describe the complex object appearance. tion are updated during tracking by removing old parts that exhibit signs of drifting and adding new ones for easy accommodation of This paper has been recommended for acceptance by Eklundh. appearance changes.Feature-based trackers often represent the Corresponding author. target by a set of sparse local features such as SIFT [7]and affine E-mail address:ywguo@nju.edu.cn (Y.Guo). invariant point detectors [8]which are often invariant to changes 1077-3142/$-see front matter2013 Elsevier Inc.All rights reserved. http://dx.doi.org/10.1016/j.cviu.2013.09.007Object tracking using learned feature manifolds q Yanwen Guo a,⇑ , Ye Chen a , Feng Tang b , Ang Li c , Weitao Luo a , Mingming Liu a aNational Key Lab for Novel Software Technology, Nanjing University, Nanjing 210023, PR China b Hewlett-Packard Laboratories, Palo Alto, CA 94304, USA cUniversity of Maryland, College Park, MD 20740, USA article info Article history: Received 10 August 2012 Accepted 30 September 2013 Available online 17 October 2013 Keywords: Feature manifold SIFT Tracking abstract Local feature based object tracking approaches have been promising in solving the tracking problems such as occlusions and illumination variations. However, existing approaches typically model feature variations using prototypes, and this discrete representation cannot capture the gradual changing prop￾erty of local appearance. In this paper, we propose to model each local feature as a feature manifold to characterize the smooth changing behavior of the feature descriptor. The manifold is constructed from a series of transformed images simulating possible variations of the feature being tracked. We propose to build a collection of linear subspaces which approximate the original manifold as a low dimensional representation. This representation is used for object tracking. Object location is located by a feature￾to-manifold matching process. Our tracking method can update the manifold status, add new feature manifolds and remove expiring ones adaptively according to object appearance. We show both qualita￾tively and quantitatively this representation significantly improves the tracking performance under occlusions and appearance variations using standard tracking dataset. 2013 Elsevier Inc. All rights reserved. 1. Introduction Object tracking is a central problem in computer vision with many applications, such as activity analysis, automated surveil￾lance, traffic monitoring, and human-computer interaction. It is essentially the problem of finding the most likely estimate of the object state given a sequence of observations. Object tracking is challenging because of: Complex object appearance. The object may have complicated appearance which is hard to model. Furthermore, it may undergo significant changes due to the pose and scale variations as well as non-rigid object motions. Occlusions. The object may be occluded by the background or other moving objects, making it difficult to be localized. Complex object motion. This is caused by either the moving pat￾tern of the object or by camera motion accompanied by object motion. There are two key components in an object tracking algorithm: object representation and dynamics. Object representation tries to model the object as accurately as possible so that the tracking algorithm can accurately describe the complex object appearance. Object dynamics model how the object appearance evolves over time to be able to handle appearance variations. The two problems are usually coupled together: the object representation should be designed to be easily updated to model appearance variations, while the object dynamics should be able to take advantage of the characteristics of object representation for model update. Traditional methods for representing the object, such as global histogram based approach in meanshift tracking [1] and PCA sub￾space based approach in EigenTracking [2], are global approaches which describe the object to be tracked as a whole. Such methods work well in many practical applications, but have several intrinsic limitations. First, it is usually very difficult for a global representa￾tion to capture local details and as a result unable to model com￾plex appearances. Second, global representations are not robust to partial occlusion. Once the object is occluded, the whole feature vector of object representation is affected. Third, global representa￾tions are hard to update. Recently, local representations have opened a promising direc￾tion to solve these problems by representing an object as a set of local parts or sparse local features. Part-based trackers generally use sets of connected or global visual properties incorporated local parts or components [3–6]. The parts used for object representa￾tion are updated during tracking by removing old parts that exhibit signs of drifting and adding new ones for easy accommodation of appearance changes. Feature-based trackers often represent the target by a set of sparse local features such as SIFT [7] and affine invariant point detectors [8] which are often invariant to changes 1077-3142/$ - see front matter 2013 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.cviu.2013.09.007 q This paper has been recommended for acceptance by J.-O. Eklundh. ⇑ Corresponding author. E-mail address: ywguo@nju.edu.cn (Y. Guo). Computer Vision and Image Understanding 118 (2014) 128–139 Contents lists available at ScienceDirect Computer Vision and Image Understanding journal homepage: www.elsevier.com/locate/cviu