《机器学习 Machine Learning》课程教学资源（书籍文献）Real-Time Human Pose Recognition in Parts from Single Depth Images

Real-Time Human Pose Recognition in Parts from Single Depth Images Jamie Shotton Andrew Fitzgibbon Mat Cook Toby Sharp Mark Finocchio Richard Moore Alex Kipman Andrew Blake Microsoft Research Cambridge Xbox Incubation Abstract We propose a new method to quickly and accurately pre- dict 3D positions of body joints from a single depth image, using no temporal information.We take an object recog- nition approach,designing an intermediate body parts rep- ont resentation that maps the difficult pose estimation problem into a simpler per-pixel classification problem.Our large and highly varied training dataset allows the classifier to estimate body parts invariant to pose,body shape,clothing, etc.Finally we generate confidence-scored 3D proposals of several body joints by reprojecting the classification result front and finding local modes. The system runs at 200 frames per second on consumer depth image body parts ◆ 3D joint proposals hardware.Our evaluation shows high accuracy on both Figure 1.Overview.From an single input depth image,a per-pixel synthetic and real test sets,and investigates the effect of sev- body part distribution is inferred.(Colors indicate the most likely eral training parameters.We achieve state of the art accu- part labels at each pixel,and correspond in the joint proposals). racy in our comparison with related work and demonstrate Local modes of this signal are estimated to give high-quality pro- improved generalization over exact whole-skeleton nearest posals for the 3D locations of body joints,even for multiple users. neighbor matching. joints of interest.Reprojecting the inferred parts into world 1.Introduction space,we localize spatial modes of each part distribution Robust interactive human body tracking has applica- and thus generate (possibly several)confidence-weighted tions including gaming,human-computer interaction,secu- proposals for the 3D locations of each skeletal joint. rity,telepresence,and even health-care.The task has re- We treat the segmentation into body parts as a per-pixel cently been greatly simplified by the introduction of real- classification task(no pairwise terms or CRF have proved time depth cameras [16,19,44,37,28,13].However,even necessary).Evaluating each pixel separately avoids a com- the best existing systems still exhibit limitations.In partic- binatorial search over the different body joints,although ular,until the launch of Kinect [21],none ran at interactive within a single part there are of course still dramatic dif- rates on consumer hardware while handling a full range of ferences in the contextual appearance.For training data, human body shapes and sizes undergoing general body mo- we generate realistic synthetic depth images of humans of tions.Some systems achieve high speeds by tracking from many shapes and sizes in highly varied poses sampled from frame to frame but struggle to re-initialize quickly and so a large motion capture database.We train a deep ran- are not robust.In this paper,we focus on pose recognition domized decision forest classifier which avoids overfitting in parts:detecting from a single depth image a small set of by using hundreds of thousands of training images.Sim- 3D position candidates for each skeletal joint.Our focus on ple,discriminative depth comparison image features yield per-frame initialization and recovery is designed to comple- 3D translation invariance while maintaining high computa- ment any appropriate tracking algorithm [7,39,16,42,13] tional efficiency.For further speed,the classifier can be run that might further incorporate temporal and kinematic co- in parallel on each pixel on a GPU [34].Finally,spatial herence.The algorithm presented here forms a core com- modes of the inferred per-pixel distributions are computed ponent of the Kinect gaming platform [21]. using mean shift [10]resulting in the 3D joint proposals. Illustrated in Fig.1 and inspired by recent object recog- An optimized implementation of our algorithm runs in nition work that divides objects into parts (e.g.[12,43]), under 5ms per frame (200 frames per second)on the Xbox our approach is driven by two key design goals:computa- 360 GPU,at least one order of magnitude faster than exist- tional efficiency and robustness.A single input depth image ing approaches.It works frame-by-frame across dramati- is segmented into a dense probabilistic body part labeling, cally differing body shapes and sizes,and the learned dis- with the parts defined to be spatially localized near skeletal criminative approach naturally handles self-occlusions and

Real-Time Human Pose Recognition in Parts from Single Depth Images Jamie Shotton Andrew Fitzgibbon Mat Cook Toby Sharp Mark Finocchio Richard Moore Alex Kipman Andrew Blake Microsoft Research Cambridge & Xbox Incubation Abstract We propose a new method to quickly and accurately pre￾dict 3D positions of body joints from a single depth image, using no temporal information. We take an object recog￾nition approach, designing an intermediate body parts rep￾resentation that maps the difficult pose estimation problem into a simpler per-pixel classification problem. Our large and highly varied training dataset allows the classifier to estimate body parts invariant to pose, body shape, clothing, etc. Finally we generate confidence-scored 3D proposals of several body joints by reprojecting the classification result and finding local modes. The system runs at 200 frames per second on consumer hardware. Our evaluation shows high accuracy on both synthetic and real test sets, and investigates the effect of sev￾eral training parameters. We achieve state of the art accu￾racy in our comparison with related work and demonstrate improved generalization over exact whole-skeleton nearest neighbor matching. 1. Introduction Robust interactive human body tracking has applica￾tions including gaming, human-computer interaction, secu￾rity, telepresence, and even health-care. The task has re￾cently been greatly simplified by the introduction of real￾time depth cameras [16, 19, 44, 37, 28, 13]. However, even the best existing systems still exhibit limitations. In partic￾ular, until the launch of Kinect [21], none ran at interactive rates on consumer hardware while handling a full range of human body shapes and sizes undergoing general body mo￾tions. Some systems achieve high speeds by tracking from frame to frame but struggle to re-initialize quickly and so are not robust. In this paper, we focus on pose recognition in parts: detecting from a single depth image a small set of 3D position candidates for each skeletal joint. Our focus on per-frame initialization and recovery is designed to comple￾ment any appropriate tracking algorithm [7, 39, 16, 42, 13] that might further incorporate temporal and kinematic co￾herence. The algorithm presented here forms a core com￾ponent of the Kinect gaming platform [21]. Illustrated in Fig. 1 and inspired by recent object recog￾nition work that divides objects into parts (e.g. [12, 43]), our approach is driven by two key design goals: computa￾tional efficiency and robustness. A single input depth image is segmented into a dense probabilistic body part labeling, with the parts defined to be spatially localized near skeletal CVPR Teaser seq 1: frame 15 seq 2: frame 236 seq 5: take 1, 72 depth image body parts 3D joint proposals Figure 1. Overview. From an single input depth image, a per-pixel body part distribution is inferred. (Colors indicate the most likely part labels at each pixel, and correspond in the joint proposals). Local modes of this signal are estimated to give high-quality pro￾posals for the 3D locations of body joints, even for multiple users. joints of interest. Reprojecting the inferred parts into world space, we localize spatial modes of each part distribution and thus generate (possibly several) confidence-weighted proposals for the 3D locations of each skeletal joint. We treat the segmentation into body parts as a per-pixel classification task (no pairwise terms or CRF have proved necessary). Evaluating each pixel separately avoids a com￾binatorial search over the different body joints, although within a single part there are of course still dramatic dif￾ferences in the contextual appearance. For training data, we generate realistic synthetic depth images of humans of many shapes and sizes in highly varied poses sampled from a large motion capture database. We train a deep ran￾domized decision forest classifier which avoids overfitting by using hundreds of thousands of training images. Sim￾ple, discriminative depth comparison image features yield 3D translation invariance while maintaining high computa￾tional efficiency. For further speed, the classifier can be run in parallel on each pixel on a GPU [34]. Finally, spatial modes of the inferred per-pixel distributions are computed using mean shift [10] resulting in the 3D joint proposals. An optimized implementation of our algorithm runs in under 5ms per frame (200 frames per second) on the Xbox 360 GPU, at least one order of magnitude faster than exist￾ing approaches. It works frame-by-frame across dramati￾cally differing body shapes and sizes, and the learned dis￾criminative approach naturally handles self-occlusions and 1

poses cropped by the image frame.We evaluate on both real matched by parameter sensitive hashing.Agarwal Triggs and synthetic depth images,containing challenging poses of [1]learn a regression from kernelized image silhouettes fea- a varied set of subjects.Even without exploiting temporal tures to pose.Sigal et al.[39]use eigen-appearance tem- or kinematic constraints,the 3D joint proposals are both ac- plate detectors for head,upper arms and lower legs pro- curate and stable.We investigate the effect of several train- posals.Felzenszwalb Huttenlocher [11]apply pictorial ing parameters and show how very deep trees can still avoid structures to estimate pose efficiently.Navaratnam et al. overfitting due to the large training set.We demonstrate [25]use the marginal statistics of unlabeled data to im- that our part proposals generalize at least as well as exact prove pose estimation.Urtasun Darrel [41]proposed a nearest-neighbor in both an idealized and realistic setting, local mixture of Gaussian Processes to regress human pose. and show a substantial improvement over the state of the Auto-context was used in [40]to obtain a coarse body part art.Further,results on silhouette images suggest more gen- labeling but this was not defined to localize joints and clas- eral applicability of our approach. sifying each frame took about 40 seconds.Rogez et al.[32] Our main contribution is to treat pose estimation as ob- train randomized decision forests on a hierarchy of classes ject recognition using a novel intermediate body parts rep- defined on a torus of cyclic human motion patterns and cam- resentation designed to spatially localize joints of interest era angles.Wang Popovic [42]track a hand clothed in a at low computational cost and high accuracy.Our experi- colored glove.Our system could be seen as automatically ments also carry several insights:(i)synthetic depth train- inferring the colors of an virtual colored suit from a depth ing data is an excellent proxy for real data;(ii)scaling up image.Bourdev Malik [6]present 'poselets'that form the learning problem with varied synthetic data is important tight clusters in both 3D pose and 2D image appearance. for high accuracy;and(iii)our parts-based approach gener- detectable using SVMs. alizes better than even an oracular exact nearest neighbor. 2.Data Related Work.Human pose estimation has generated a vast literature(surveyed in [22.29)).The recent availability Pose estimation research has often focused on techniques of depth cameras has spurred further progress [16,19.28]. to overcome lack of training data [25],because of two prob- Grest et al.[16]use Iterated Closest Point to track a skele- lems.First,generating realistic intensity images using com- ton of a known size and starting position.Anguelov et al. puter graphics techniques [33,27,26]is hampered by the [3]segment puppets in 3D range scan data into head,limbs, huge color and texture variability induced by clothing,hair, torso,and background using spin images and a MRF.In and skin,often meaning that the data are reduced to 2D sil- [44.Zhu Fujimura build heuristic detectors for coarse houettes [1].Although depth cameras significantly reduce upper body parts(head,torso,arms)using a linear program- this difficulty,considerable variation in body and clothing ming relaxation,but require a T-pose initialization to size shape remains.The second limitation is that synthetic body the model.Siddiqui Medioni [37]hand craft head,hand, pose images are of necessity fed by motion-capture(mocap) and forearm detectors,and show data-driven MCMC model data.Although techniques exist to simulate human motion fitting outperforms ICP.Kalogerakis et al.[18]classify and (e.g.[38])they do not yet produce the range of volitional segment vertices in a full closed 3D mesh into different motions of a human subject. parts,but do not deal with occlusions and are sensitive to In this section we review depth imaging and show how mesh topology.Most similar to our approach,Plagemann we use real mocap data,retargetted to a variety of base char- et al.[28]build a 3D mesh to find geodesic extrema inter- acter models,to synthesize a large,varied dataset.We be- est points which are classified into 3 parts:head,hand,and lieve this dataset to considerably advance the state of the art foot.Their method provides both a location and orientation in both scale and variety,and demonstrate the importance estimate of these parts,but does not distinguish left from of such a large dataset in our evaluation. right and the use of interest points limits the choice of parts. 2.1.Depth imaging Advances have also been made using conventional in- Depth imaging technology has advanced dramatically tensity cameras,though typically at much higher computa- over the last few years,finally reaching a consumer price tional cost.Bregler Malik [7]track humans using twists point with the launch of Kinect[21].Pixels in a depth image and exponential maps from a known initial pose.Ioffe indicate calibrated depth in the scene,rather than a measure Forsyth [17]group parallel edges as candidate body seg- of intensity or color.We employ the Kinect camera which ments and prune combinations of segments using a pro- gives a 640x480 image at 30 frames per second with depth jected classifier.Mori Malik [24]use the shape con- resolution of a few centimeters. text descriptor to match exemplars.Ramanan Forsyth Depth cameras offer several advantages over traditional [31]find candidate body segments as pairs of parallel lines, intensity sensors,working in low light levels,giving a cali- clustering appearances across frames.Shakhnarovich et al. brated scale estimate,being color and texture invariant,and [33]estimate upper body pose,interpolating k-NN poses resolving silhouette ambiguities in pose.They also greatly

poses cropped by the image frame. We evaluate on both real and synthetic depth images, containing challenging poses of a varied set of subjects. Even without exploiting temporal or kinematic constraints, the 3D joint proposals are both ac￾curate and stable. We investigate the effect of several train￾ing parameters and show how very deep trees can still avoid overfitting due to the large training set. We demonstrate that our part proposals generalize at least as well as exact nearest-neighbor in both an idealized and realistic setting, and show a substantial improvement over the state of the art. Further, results on silhouette images suggest more gen￾eral applicability of our approach. Our main contribution is to treat pose estimation as ob￾ject recognition using a novel intermediate body parts rep￾resentation designed to spatially localize joints of interest at low computational cost and high accuracy. Our experi￾ments also carry several insights: (i) synthetic depth train￾ing data is an excellent proxy for real data; (ii) scaling up the learning problem with varied synthetic data is important for high accuracy; and (iii) our parts-based approach gener￾alizes better than even an oracular exact nearest neighbor. Related Work. Human pose estimation has generated a vast literature (surveyed in [22, 29]). The recent availability of depth cameras has spurred further progress [16, 19, 28]. Grest et al. [16] use Iterated Closest Point to track a skele￾ton of a known size and starting position. Anguelov et al. [3] segment puppets in 3D range scan data into head, limbs, torso, and background using spin images and a MRF. In [44], Zhu & Fujimura build heuristic detectors for coarse upper body parts (head, torso, arms) using a linear program￾ming relaxation, but require a T-pose initialization to size the model. Siddiqui & Medioni [37] hand craft head, hand, and forearm detectors, and show data-driven MCMC model fitting outperforms ICP. Kalogerakis et al. [18] classify and segment vertices in a full closed 3D mesh into different parts, but do not deal with occlusions and are sensitive to mesh topology. Most similar to our approach, Plagemann et al. [28] build a 3D mesh to find geodesic extrema inter￾est points which are classified into 3 parts: head, hand, and foot. Their method provides both a location and orientation estimate of these parts, but does not distinguish left from right and the use of interest points limits the choice of parts. Advances have also been made using conventional in￾tensity cameras, though typically at much higher computa￾tional cost. Bregler & Malik [7] track humans using twists and exponential maps from a known initial pose. Ioffe & Forsyth [17] group parallel edges as candidate body seg￾ments and prune combinations of segments using a pro￾jected classifier. Mori & Malik [24] use the shape con￾text descriptor to match exemplars. Ramanan & Forsyth [31] find candidate body segments as pairs of parallel lines, clustering appearances across frames. Shakhnarovich et al. [33] estimate upper body pose, interpolating k-NN poses matched by parameter sensitive hashing. Agarwal & Triggs [1] learn a regression from kernelized image silhouettes fea￾tures to pose. Sigal et al. [39] use eigen-appearance tem￾plate detectors for head, upper arms and lower legs pro￾posals. Felzenszwalb & Huttenlocher [11] apply pictorial structures to estimate pose efficiently. Navaratnam et al. [25] use the marginal statistics of unlabeled data to im￾prove pose estimation. Urtasun & Darrel [41] proposed a local mixture of Gaussian Processes to regress human pose. Auto-context was used in [40] to obtain a coarse body part labeling but this was not defined to localize joints and clas￾sifying each frame took about 40 seconds. Rogez et al. [32] train randomized decision forests on a hierarchy of classes defined on a torus of cyclic human motion patterns and cam￾era angles. Wang & Popovic [ ´ 42] track a hand clothed in a colored glove. Our system could be seen as automatically inferring the colors of an virtual colored suit from a depth image. Bourdev & Malik [6] present ‘poselets’ that form tight clusters in both 3D pose and 2D image appearance, detectable using SVMs. 2. Data Pose estimation research has often focused on techniques to overcome lack of training data [25], because of two prob￾lems. First, generating realistic intensity images using com￾puter graphics techniques [33, 27, 26] is hampered by the huge color and texture variability induced by clothing, hair, and skin, often meaning that the data are reduced to 2D sil￾houettes [1]. Although depth cameras significantly reduce this difficulty, considerable variation in body and clothing shape remains. The second limitation is that synthetic body pose images are of necessity fed by motion-capture (mocap) data. Although techniques exist to simulate human motion (e.g. [38]) they do not yet produce the range of volitional motions of a human subject. In this section we review depth imaging and show how we use real mocap data, retargetted to a variety of base char￾acter models, to synthesize a large, varied dataset. We be￾lieve this dataset to considerably advance the state of the art in both scale and variety, and demonstrate the importance of such a large dataset in our evaluation. 2.1. Depth imaging Depth imaging technology has advanced dramatically over the last few years, finally reaching a consumer price point with the launch of Kinect [21]. Pixels in a depth image indicate calibrated depth in the scene, rather than a measure of intensity or color. We employ the Kinect camera which gives a 640x480 image at 30 frames per second with depth resolution of a few centimeters. Depth cameras offer several advantages over traditional intensity sensors, working in low light levels, giving a cali￾brated scale estimate, being color and texture invariant, and resolving silhouette ambiguities in pose. They also greatly

鲁游大冷棉 Figure 2.Synthetic and real data.Pairs of depth image and ground truth body parts.Note wide variety in pose,shape,clothing.and crop. simplify the task of background subtraction which we as- the appearance variations we hope to recognize at test time. sume in this work.But most importantly for our approach, While depth/scale and translation variations are handled ex- it is straightforward to synthesize realistic depth images of plicitly in our features(see below).other invariances cannot people and thus build a large training dataset cheaply. be encoded efficiently.Instead we learn invariance from the 2.2.Motion capture data data to camera pose,body pose,and body size and shape. The synthesis pipeline first randomly samples a set of The human body is capable of an enormous range of parameters,and then uses standard computer graphics tech- poses which are difficult to simulate.Instead,we capture a niques to render depth and(see below)body part images large database of motion capture(mocap)of human actions. from texture mapped 3D meshes.The mocap is retarget- Our aim was to span the wide variety of poses people would ting to each of 15 base meshes spanning the range of body make in an entertainment scenario.The database consists of shapes and sizes,using [4].Further slight random vari- approximately 500k frames in a few hundred sequences of ation in height and weight give extra coverage of body driving,dancing,kicking,running,navigating menus,etc. shapes.Other randomized parameters include the mocap We expect our semi-local body part classifier to gener- frame,camera pose,camera noise,clothing and hairstyle. alize somewhat to unseen poses.In particular,we need not We provide more details of these variations in the supple- record all possible combinations of the different limbs;in mentary material.Fig.2 compares the varied output of the practice,a wide range of poses proves sufficient.Further, pipeline to hand-labeled real camera images. we need not record mocap with variation in rotation about the vertical axis,mirroring left-right,scene position,body 3.Body Part Inference and Joint Proposals shape and size,or camera pose,all of which can be added In this section we describe our intermediate body parts in (semi-)automatically. representation,detail the discriminative depth image fea- Since the classifier uses no temporal information,we tures,review decision forests and their application to body are interested only in static poses and not motion.Often, part recognition,and finally discuss how a mode finding al- changes in pose from one mocap frame to the next are so gorithm is used to generate joint position proposals. small as to be insignificant.We thus discard many similar, redundant poses from the initial mocap data using 'furthest 3.1.Body part labeling neighbor'clustering [15]where the distance between poses A key contribution of this work is our intermediate body pi and p2 is defined as maxjpp2,the maximum Eu- part representation.We define several localized body part clidean distance over body joints j.We use a subset of 100k labels that densely cover the body,as color-coded in Fig.2. poses such that no two poses are closer than 5cm. Some of these parts are defined to directly localize partic- We have found it necessary to iterate the process of mo- ular skeletal joints of interest,while others fill the gaps or tion capture,sampling from our model,training the classi- could be used in combination to predict other joints.Our in- fier,and testing joint prediction accuracy in order to refine termediate representation transforms the problem into one the mocap database with regions of pose space that had been that can readily be solved by efficient classification algo- previously missed out.Our early experiments employed rithms;we show in Sec.4.3 that the penalty paid for this the CMU mocap database [9]which gave acceptable results transformation is small. though covered far less of pose space. The parts are specified in a texture map that is retargetted to skin the various characters during rendering.The pairs of 2.3.Generating synthetic data depth and body part images are used as fully labeled data for We build a randomized rendering pipeline from which learning the classifier(see below).For the experiments in we can sample fully labeled training images.Our goals in this paper,we use 31 body parts:LU/RU/LW/RW head,neck, building this pipeline were twofold:realism and variety.For L/R shoulder,LU/RU/LW/RW arm,L/R elbow,L/R wrist,L/R the learned model to work well,the samples must closely hand,LU/RU/LW/RW torso,LU/RU/LW/RW leg,L/R knee, resemble real camera images,and contain good coverage of L/R ankle,L/R foot (Left,Right,Upper,lower).Distinct

Training & Test Data synthetic (train & test) real (test) synthetic (train & test) real (test) Figure 2. Synthetic and real data. Pairs of depth image and ground truth body parts. Note wide variety in pose, shape, clothing, and crop. simplify the task of background subtraction which we as￾sume in this work. But most importantly for our approach, it is straightforward to synthesize realistic depth images of people and thus build a large training dataset cheaply. 2.2. Motion capture data The human body is capable of an enormous range of poses which are difficult to simulate. Instead, we capture a large database of motion capture (mocap) of human actions. Our aim was to span the wide variety of poses people would make in an entertainment scenario. The database consists of approximately 500k frames in a few hundred sequences of driving, dancing, kicking, running, navigating menus, etc. We expect our semi-local body part classifier to gener￾alize somewhat to unseen poses. In particular, we need not record all possible combinations of the different limbs; in practice, a wide range of poses proves sufficient. Further, we need not record mocap with variation in rotation about the vertical axis, mirroring left-right, scene position, body shape and size, or camera pose, all of which can be added in (semi-)automatically. Since the classifier uses no temporal information, we are interested only in static poses and not motion. Often, changes in pose from one mocap frame to the next are so small as to be insignificant. We thus discard many similar, redundant poses from the initial mocap data using ‘furthest neighbor’ clustering [15] where the distance between poses p1 and p2 is defined as maxj kp j 1 −p j 2 k2, the maximum Eu￾clidean distance over body joints j. We use a subset of 100k poses such that no two poses are closer than 5cm. We have found it necessary to iterate the process of mo￾tion capture, sampling from our model, training the classi- fier, and testing joint prediction accuracy in order to refine the mocap database with regions of pose space that had been previously missed out. Our early experiments employed the CMU mocap database [9] which gave acceptable results though covered far less of pose space. 2.3. Generating synthetic data We build a randomized rendering pipeline from which we can sample fully labeled training images. Our goals in building this pipeline were twofold: realism and variety. For the learned model to work well, the samples must closely resemble real camera images, and contain good coverage of the appearance variations we hope to recognize at test time. While depth/scale and translation variations are handled ex￾plicitly in our features (see below), other invariances cannot be encoded efficiently. Instead we learn invariance from the data to camera pose, body pose, and body size and shape. The synthesis pipeline first randomly samples a set of parameters, and then uses standard computer graphics tech￾niques to render depth and (see below) body part images from texture mapped 3D meshes. The mocap is retarget￾ting to each of 15 base meshes spanning the range of body shapes and sizes, using [4]. Further slight random vari￾ation in height and weight give extra coverage of body shapes. Other randomized parameters include the mocap frame, camera pose, camera noise, clothing and hairstyle. We provide more details of these variations in the supple￾mentary material. Fig. 2 compares the varied output of the pipeline to hand-labeled real camera images. 3. Body Part Inference and Joint Proposals In this section we describe our intermediate body parts representation, detail the discriminative depth image fea￾tures, review decision forests and their application to body part recognition, and finally discuss how a mode finding al￾gorithm is used to generate joint position proposals. 3.1. Body part labeling A key contribution of this work is our intermediate body part representation. We define several localized body part labels that densely cover the body, as color-coded in Fig. 2. Some of these parts are defined to directly localize partic￾ular skeletal joints of interest, while others fill the gaps or could be used in combination to predict other joints. Our in￾termediate representation transforms the problem into one that can readily be solved by efficient classification algo￾rithms; we show in Sec. 4.3 that the penalty paid for this transformation is small. The parts are specified in a texture map that is retargetted to skin the various characters during rendering. The pairs of depth and body part images are used as fully labeled data for learning the classifier (see below). For the experiments in this paper, we use 31 body parts: LU/RU/LW/RW head, neck, L/R shoulder, LU/RU/LW/RW arm, L/R elbow, L/R wrist, L/R hand, LU/RU/LW/RW torso, LU/RU/LW/RW leg, L/R knee, L/R ankle, L/R foot (Left, Right, Upper, loWer). Distinct

(1,3 (1,x) tree 1 tree 2 P(C Figure 3.Depth image features.The yellow crosses indicates the Figure 4.Randomized Decision Forests.A forest is an ensemble pixel x being classified.The red circles indicate the offset pixels of trees.Each tree consists of split nodes (blue)and leaf nodes as defined in Eq.1.In (a),the two example features give a large (green).The red arrows indicate the different paths that might be depth difference response.In (b),the same two features at new taken by different trees for a particular input. image locations give a much smaller response. 3.3.Randomized decision forests parts for left and right allow the classifier to disambiguate Randomized decision trees and forests [35,30.2.8]have the left and right sides of the body. proven fast and effective multi-class classifiers for many Of course,the precise definition of these parts could be tasks [20,23,36],and can be implemented efficiently on the changed to suit a particular application.For example,in an GPU [34].As illustrated in Fig.4,a forest is an ensemble upper body tracking scenario,all the lower body parts could of T decision trees,each consisting of split and leaf nodes. be merged.Parts should be sufficiently small to accurately Each split node consists of a feature fe and a threshold T. localize body joints,but not too numerous as to waste ca- To classify pixel x in image I.one starts at the root and re- pacity of the classifier. peatedly evaluates Eq.1,branching left or right according 3.2.Depth image features to the comparison to threshold T.At the leaf node reached We employ simple depth comparison features,inspired in tree t,a learned distribution P(cI,x)over body part la- by those in [20].At a given pixel x,the features compute bels c is stored.The distributions are averaged together for all trees in the forest to give the final classification u=(+)-d(+) (1 T P(cl,x)= 1>P(dI,x)· (2) where d(x)is the depth at pixel x in image 1,and parame- t=1 ters =(u.v)describe offsets u and v.The normalization Training.Each tree is trained on a different set of randomly of the offsets by ensures the features are depth invari- synthesized images.A random subset of 2000 example pix- ant:at a given point on the body,a fixed world space offset els from each image is chosen to ensure a roughly even dis- will result whether the pixel is close or far from the camera. tribution across body parts.Each tree is trained using the The features are thus 3D translation invariant(modulo per- following algorithm [20]: spective effects).If an offset pixel lies on the background 1.Randomly propose a set of splitting candidates= or outside the bounds of the image,the depth probe d(x') (0,T)(feature parameters 0 and thresholds T). is given a large positive constant value. Fig.3 illustrates two features at different pixel locations 2.Partition the set of examples Q={(I,x)}into left x.Feature fe,looks upwards:Eq.1 will give a large pos- and right subsets by each o: itive response for pixels x near the top of the body,but a Q(o)={(I,x)1f(I,x)<T} (3) value close to zero for pixels x lower down the body.Fea- Q.(o)=Q\Q(⊙) (4) ture fo may instead help find thin vertical structures such as the arm. 3.Compute the o giving the largest gain in information: Individually these features provide only a weak signal about which part of the body the pixel belongs to,but in =argmax G(o) (5) combination in a decision forest they are sufficient to accu- rately disambiguate all trained parts.The design of these G(o) =H(Q)- 1Q.(o刨H(Q.()(6 features was strongly motivated by their computational effi- sEfL,r 121 ciency:no preprocessing is needed;each feature need only read at most 3 image pixels and perform at most 5 arithmetic where Shannon entropy H(Q)is computed on the nor- operations;and the features can be straightforwardly imple- malized histogram of body part labels l(x)for all mented on the GPU.Given a larger computational budget, (I,x)∈Q one could employ potentially more powerful features based 4.If the largest gain G(*)is sufficient,and the depth in on,for example,depth integrals over regions,curvature,or the tree is below a maximum,then recurse for left and local descriptors e.g.[5]. right subsets (*and ()

(a) body parts Image Features (b) 𝜃2 𝜃1 𝜃2 𝜃2 𝜃1 𝜃2 Figure 3. Depth image features. The yellow crosses indicates the pixel x being classified. The red circles indicate the offset pixels as defined in Eq. 1. In (a), the two example features give a large depth difference response. In (b), the same two features at new image locations give a much smaller response. parts for left and right allow the classifier to disambiguate the left and right sides of the body. Of course, the precise definition of these parts could be changed to suit a particular application. For example, in an upper body tracking scenario, all the lower body parts could be merged. Parts should be sufficiently small to accurately localize body joints, but not too numerous as to waste ca￾pacity of the classifier. 3.2. Depth image features We employ simple depth comparison features, inspired by those in [20]. At a given pixel x, the features compute fθ(I, x) = dI  x + u dI (x)  − dI  x + v dI (x)  , (1) where dI (x) is the depth at pixel x in image I, and parame￾ters θ = (u, v) describe offsets u and v. The normalization of the offsets by 1 dI (x) ensures the features are depth invari￾ant: at a given point on the body, a fixed world space offset will result whether the pixel is close or far from the camera. The features are thus 3D translation invariant (modulo per￾spective effects). If an offset pixel lies on the background or outside the bounds of the image, the depth probe dI (x 0 ) is given a large positive constant value. Fig. 3 illustrates two features at different pixel locations x. Feature fθ1 looks upwards: Eq. 1 will give a large pos￾itive response for pixels x near the top of the body, but a value close to zero for pixels x lower down the body. Fea￾ture fθ2 may instead help find thin vertical structures such as the arm. Individually these features provide only a weak signal about which part of the body the pixel belongs to, but in combination in a decision forest they are sufficient to accu￾rately disambiguate all trained parts. The design of these features was strongly motivated by their computational effi- ciency: no preprocessing is needed; each feature need only read at most 3 image pixels and perform at most 5 arithmetic operations; and the features can be straightforwardly imple￾mented on the GPU. Given a larger computational budget, one could employ potentially more powerful features based on, for example, depth integrals over regions, curvature, or local descriptors e.g. [5]. Random Forests … tree 1 tree 𝑇 (𝐼, x) (𝐼, x) 𝑃𝑇(𝑐) 𝑃1(𝑐) Figure 4. Randomized Decision Forests. A forest is an ensemble of trees. Each tree consists of split nodes (blue) and leaf nodes (green). The red arrows indicate the different paths that might be taken by different trees for a particular input. 3.3. Randomized decision forests Randomized decision trees and forests [35, 30, 2, 8] have proven fast and effective multi-class classifiers for many tasks [20, 23, 36], and can be implemented efficiently on the GPU [34]. As illustrated in Fig. 4, a forest is an ensemble of T decision trees, each consisting of split and leaf nodes. Each split node consists of a feature fθ and a threshold τ . To classify pixel x in image I, one starts at the root and re￾peatedly evaluates Eq. 1, branching left or right according to the comparison to threshold τ . At the leaf node reached in tree t, a learned distribution Pt(c|I, x) over body part la￾bels c is stored. The distributions are averaged together for all trees in the forest to give the final classification P(c|I, x) = 1 T X T t=1 Pt(c|I, x) . (2) Training. Each tree is trained on a different set of randomly synthesized images. A random subset of 2000 example pix￾els from each image is chosen to ensure a roughly even dis￾tribution across body parts. Each tree is trained using the following algorithm [20]: 1. Randomly propose a set of splitting candidates φ = (θ, τ ) (feature parameters θ and thresholds τ ). 2. Partition the set of examples Q = {(I, x)} into left and right subsets by each φ: Ql(φ) = { (I, x) | fθ(I, x) < τ } (3) Qr(φ) = Q \ Ql(φ) (4) 3. Compute the φ giving the largest gain in information: φ ? = argmax φ G(φ) (5) G(φ) = H(Q) − X s∈{l,r} |Qs(φ)| |Q| H(Qs(φ)) (6) where Shannon entropy H(Q) is computed on the nor￾malized histogram of body part labels lI (x) for all (I, x) ∈ Q. 4. If the largest gain G(φ ? ) is sufficient, and the depth in the tree is below a maximum, then recurse for left and right subsets Ql(φ ? ) and Qr(φ ? )

Figure 5.Example inferences.Synthetic(top row);real (middle):failure modes(bottom).Left column:ground truth for a neutral pose as a reference.In each example we see the depth image,the inferred most likely body part labels,and the joint proposals show as front,right, and top views (overlaid on a depth point cloud).Only the most confident proposal for each joint above a fixed,shared threshold is shown. To keep the training times down we employ a distributed The detected modes lie on the surface of the body.Each implementation.Training 3 trees to depth 20 from 1 million mode is therefore pushed back into the scene by a learned images takes about a day on a 1000 core cluster. z offset Ce to produce a final joint position proposal.This 3.4.Joint position proposals simple,efficient approach works well in practice.The band- Body part recognition as described above infers per-pixel widths be,probability threshold Ac,and surface-to-interior z offset C are optimized per-part on a hold-out validation information.This information must now be pooled across pixels to generate reliable proposals for the positions of 3D set of 5000 images by grid search.(As an indication,this resulted in mean bandwidth 0.065m,probability threshold skeletal joints.These proposals are the final output of our 0.14.and z offset 0.039m). algorithm,and could be used by a tracking algorithm to self- initialize and recover from failure. 4.Experiments A simple option is to accumulate the global 3D centers In this section we describe the experiments performed to of probability mass for each part,using the known cali- evaluate our method.We show both qualitative and quan- brated depth.However,outlying pixels severely degrade titative results on several challenging datasets,and com- the quality of such a global estimate.Instead we employ a pare with both nearest-neighbor approaches and the state local mode-finding approach based on mean shift [10]with of the art [13].We provide further results in the supple- a weighted Gaussian kernel. mentary material.Unless otherwise specified,parameters We define a density estimator per body part as below were set as:3 trees,20 deep,300k training images per tree,2000 training example pixels per image,2000 can- fe()>Wic exp didate features 0,and 50 candidate thresholdsr per feature Test data.We use challenging synthetic and real depth im- where x is a coordinate in 3D world space.N is the number ages to evaluate our approach.For our synthetic test set, of image pixels,wic is a pixel weighting,x;is the reprojec- we synthesize 5000 depth images,together with the ground tion of image pixel x;into world space given depth dr(xi), truth body part labels and joint positions.The original mo- and be is a learned per-part bandwidth.The pixel weighting cap poses used to generate these images are held out from wie considers both the inferred body part probability at the the training data.Our real test set consists of 8808 frames of pixel and the world surface area of the pixel: real depth images over 15 different subjects,hand-labeled wie=P(cI,xi).dI(xi)2. (8) with dense body parts and 7 upper body joint positions.We also evaluate on the real depth data from [131.The results This ensures density estimates are depth invariant and gave suggest that effects seen on synthetic data are mirrored in a small but significant improvement in joint prediction ac- the real data,and further that our synthetic test set is by far curacy.Depending on the definition of body parts,the pos- the 'hardest'due to the extreme variability in pose and body terior P(cI,x)can be pre-accumulated over a small set of shape.For most experiments we limit the rotation of the parts.For example,in our experiments the four body parts user to +120 in both training and synthetic test data since covering the head are merged to localize the head joint. the user is facing the camera(0)in our main entertainment Mean shift is used to find modes in this density effi- scenario,though we also evaluate the full 360 scenario. ciently.All pixels above a learned probability threshold Ac Error metrics.We quantify both classification and joint are used as starting points for part c.A final confidence es- prediction accuracy.For classification,we report the av- timate is given as a sum of the pixel weights reaching each erage per-class accuracy,i.e.the average of the diagonal of mode.This proved more reliable than taking the modal den- the confusion matrix between the ground truth part label and sity estimate. the most likely inferred part label.This metric weights each

• depth, map, front/right/top • pose, distances, cropping, camera angles, body size and shape (e.g. small child, thin/fat), • failure modes: underlying probability correct, can detect failures with confidence • synthetic / real / failures Example inferences Figure 5. Example inferences. Synthetic (top row); real (middle); failure modes (bottom). Left column: ground truth for a neutral pose as a reference. In each example we see the depth image, the inferred most likely body part labels, and the joint proposals show as front, right, and top views (overlaid on a depth point cloud). Only the most confident proposal for each joint above a fixed, shared threshold is shown. To keep the training times down we employ a distributed implementation. Training 3 trees to depth 20 from 1 million images takes about a day on a 1000 core cluster. 3.4. Joint position proposals Body part recognition as described above infers per-pixel information. This information must now be pooled across pixels to generate reliable proposals for the positions of 3D skeletal joints. These proposals are the final output of our algorithm, and could be used by a tracking algorithm to self￾initialize and recover from failure. A simple option is to accumulate the global 3D centers of probability mass for each part, using the known cali￾brated depth. However, outlying pixels severely degrade the quality of such a global estimate. Instead we employ a local mode-finding approach based on mean shift [10] with a weighted Gaussian kernel. We define a density estimator per body part as fc(xˆ) ∝ X N i=1 wic exp − xˆ − xˆi bc 2 ! , (7) where xˆ is a coordinate in 3D world space, N is the number of image pixels, wic is a pixel weighting, xˆi is the reprojec￾tion of image pixel xi into world space given depth dI (xi), and bc is a learned per-part bandwidth. The pixel weighting wic considers both the inferred body part probability at the pixel and the world surface area of the pixel: wic = P(c|I, xi) · dI (xi) 2 . (8) This ensures density estimates are depth invariant and gave a small but significant improvement in joint prediction ac￾curacy. Depending on the definition of body parts, the pos￾terior P(c|I, x) can be pre-accumulated over a small set of parts. For example, in our experiments the four body parts covering the head are merged to localize the head joint. Mean shift is used to find modes in this density effi- ciently. All pixels above a learned probability threshold λc are used as starting points for part c. A final confidence es￾timate is given as a sum of the pixel weights reaching each mode. This proved more reliable than taking the modal den￾sity estimate. The detected modes lie on the surface of the body. Each mode is therefore pushed back into the scene by a learned z offset ζc to produce a final joint position proposal. This simple, efficient approach works well in practice. The band￾widths bc, probability threshold λc, and surface-to-interior z offset ζc are optimized per-part on a hold-out validation set of 5000 images by grid search. (As an indication, this resulted in mean bandwidth 0.065m, probability threshold 0.14, and z offset 0.039m). 4. Experiments In this section we describe the experiments performed to evaluate our method. We show both qualitative and quan￾titative results on several challenging datasets, and com￾pare with both nearest-neighbor approaches and the state of the art [13]. We provide further results in the supple￾mentary material. Unless otherwise specified, parameters below were set as: 3 trees, 20 deep, 300k training images per tree, 2000 training example pixels per image, 2000 can￾didate features θ, and 50 candidate thresholds τ per feature. Test data. We use challenging synthetic and real depth im￾ages to evaluate our approach. For our synthetic test set, we synthesize 5000 depth images, together with the ground truth body part labels and joint positions. The original mo￾cap poses used to generate these images are held out from the training data. Our real test set consists of 8808 frames of real depth images over 15 different subjects, hand-labeled with dense body parts and 7 upper body joint positions. We also evaluate on the real depth data from [13]. The results suggest that effects seen on synthetic data are mirrored in the real data, and further that our synthetic test set is by far the ‘hardest’ due to the extreme variability in pose and body shape. For most experiments we limit the rotation of the user to ±120◦ in both training and synthetic test data since the user is facing the camera (0 ◦ ) in our main entertainment scenario, though we also evaluate the full 360◦ scenario. Error metrics. We quantify both classification and joint prediction accuracy. For classification, we report the av￾erage per-class accuracy, i.e. the average of the diagonal of the confusion matrix between the ground truth part label and the most likely inferred part label. This metric weights each

60% 60% 60% 60% Synthetic Test Set Real Test Set h一Real test data 55% 55% 55% -Synthetic test data 509% 509 50% 45% 45% SE 45% ◆一Synthetic test set 04 40% 40% 209% Real test set 35% 35% 35% -Silhouette (scale) -900k training images 900k training images -Silhouette(no scale) 30% 15k training images 15k training images 109% 30% 30% 10 1000 100000 、12 16 20 10 15 20 100 200 (a)Num.training images (log scale)(b) 309 Depth of trees Depth of trees (c) Maximum probe offset(pixel meters) Figure 6.Training parameters vs.classification accuracy.(a)Number of training images.(b)Depth of trees.(c)Maximum probe offset. body part equally despite their varying sizes,though misla- 4.2.Classification accuracy belings on the part boundaries reduce the absolute numbers. We investigate the effect of several training parameters For joint proposals,we generate recall-precision curves on classification accuracy.The trends are highly correlated as a function of confidence threshold.We quantify accuracy between the synthetic and real test sets,and the real test as average precision per joint,or mean average precision set appears consistently 'easier'than the synthetic test set, (mAP)over all joints.The first joint proposal within D me- probably due to the less varied poses present. ters of the ground truth position is taken as a true positive, Number of training images.In Fig.6(a)we show how while other proposals also within D meters count as false test accuracy increases approximately logarithmically with positives.This penalizes multiple spurious detections near the number of randomly generated training images,though the correct position which might slow a downstream track- starts to tail off around 100k images.As shown below,this ing algorithm.Any joint proposals outside D meters also saturation is likely due to the limited model capacity of a 3 count as false positives.Note that all proposals (not just the most confident)are counted in this metric.Joints invisible tree,20 deep decision forest. in the image are not penalized as false negatives. We set Silhouette images.We also show in Fig.6(a)the quality D=0.1m below,approximately the accuracy of the hand- of our approach on synthetic silhouette images,where the labeled real test data ground truth.The strong correlation features in Eg.I are either given scale (as the mean depth) of classification and joint prediction accuracy (c.f.the blue or not (a fixed constant depth).For the corresponding joint curves in Figs.6(a)and 8(a))suggests the trends observed prediction using a 2D metric with a 10 pixel true positive below for one also apply for the other. threshold,we got 0.539 mAP with scale and 0.465 mAP without.While clearly a harder task due to depth ambigui- 4.1.Qualitative results ties,these results suggest the applicability of our approach Fig.5 shows example inferences of our algorithm.Note to other imaging modalities. high accuracy of both classification and joint prediction Depth of trees.Fig.6(b)shows how the depth of trees af- across large variations in body and camera pose,depth in fects test accuracy using either 15k or 900k images.Of all scene,cropping,and body size and shape (e.g.small child the training parameters,depth appears to have the most sig- vs.heavy adult).The bottom row shows some failure modes nificant effect as it directly impacts the model capacity of of the body part classification.The first example shows the classifier.Using only 15k images we observe overfitting a failure to distinguish subtle changes in the depth image beginning around depth 17,but the enlarged 900k training such as the crossed arms.Often (as with the second and set avoids this.The high accuracy gradient at depth 20 sug- third failure examples)the most likely body part is incor- gests even better results can be achieved by training still rect,but there is still sufficient correct probability mass in deeper trees,at a small extra run-time computational cost distribution P(cI,x)that an accurate proposal can still be and a large extra memory penalty.Of practical interest is generated.The fourth example shows a failure to generalize that,until about depth 10,the training set size matters little, well to an unseen pose.but the confidence gates bad propos- suggesting an efficient training strategy. als,maintaining high precision at the expense of recall. Maximum probe offset.The range of depth probe offsets Note that no temporal or kinematic constraints (other allowed during training has a large effect on accuracy.We than those implicit in the training data)are used for any show this in Fig.6(c)for 5k training images,where 'maxi- of our results.Despite this,per-frame results on video se- mum probe offset'means the max.absolute value proposed quences in the supplementary material show almost every for both x and y coordinates of u and v in Eg.1.The con- joint accurately predicted with remarkably little jitter. centric boxes on the right show the 5 tested maximum off-

30% 35% 40% 45% 50% 55% 60% 0 100 200 300 Average per-class accuracy Maximum probe offset (pixel meters) Real test data Synthetic test data Combined Results 30% 35% 40% 45% 50% 55% 60% 5 10 15 20 Depth of trees Real Test Set 900k training images 15k training images (a) (b) (c) 10% 20% 30% 40% 50% 60% 10 1000 100000 Average per-class accuracy Num. training images (log scale) Synthetic test set Real test set Silhouette (scale) Silhouette (no scale) 30% 35% 40% 45% 50% 55% 60% 8 12 16 20 Average per-class accuracy Depth of trees Synthetic Test Set 900k training images 15k training images Figure 6. Training parameters vs. classification accuracy. (a) Number of training images. (b) Depth of trees. (c) Maximum probe offset. body part equally despite their varying sizes, though misla￾belings on the part boundaries reduce the absolute numbers. For joint proposals, we generate recall-precision curves as a function of confidence threshold. We quantify accuracy as average precision per joint, or mean average precision (mAP) over all joints.The first joint proposal within D me￾ters of the ground truth position is taken as a true positive, while other proposals also within D meters count as false positives. This penalizes multiple spurious detections near the correct position which might slow a downstream track￾ing algorithm. Any joint proposals outside D meters also count as false positives. Note that all proposals (not just the most confident) are counted in this metric. Joints invisible in the image are not penalized as false negatives. We set D = 0.1m below, approximately the accuracy of the hand￾labeled real test data ground truth. The strong correlation of classification and joint prediction accuracy (c.f . the blue curves in Figs. 6(a) and 8(a)) suggests the trends observed below for one also apply for the other. 4.1. Qualitative results Fig. 5 shows example inferences of our algorithm. Note high accuracy of both classification and joint prediction across large variations in body and camera pose, depth in scene, cropping, and body size and shape (e.g. small child vs. heavy adult). The bottom row shows some failure modes of the body part classification. The first example shows a failure to distinguish subtle changes in the depth image such as the crossed arms. Often (as with the second and third failure examples) the most likely body part is incor￾rect, but there is still sufficient correct probability mass in distribution P(c|I, x) that an accurate proposal can still be generated. The fourth example shows a failure to generalize well to an unseen pose, but the confidence gates bad propos￾als, maintaining high precision at the expense of recall. Note that no temporal or kinematic constraints (other than those implicit in the training data) are used for any of our results. Despite this, per-frame results on video se￾quences in the supplementary material show almost every joint accurately predicted with remarkably little jitter. 4.2. Classification accuracy We investigate the effect of several training parameters on classification accuracy. The trends are highly correlated between the synthetic and real test sets, and the real test set appears consistently ‘easier’ than the synthetic test set, probably due to the less varied poses present. Number of training images. In Fig. 6(a) we show how test accuracy increases approximately logarithmically with the number of randomly generated training images, though starts to tail off around 100k images. As shown below, this saturation is likely due to the limited model capacity of a 3 tree, 20 deep decision forest. Silhouette images. We also show in Fig. 6(a) the quality of our approach on synthetic silhouette images, where the features in Eq. 1 are either given scale (as the mean depth) or not (a fixed constant depth). For the corresponding joint prediction using a 2D metric with a 10 pixel true positive threshold, we got 0.539 mAP with scale and 0.465 mAP without. While clearly a harder task due to depth ambigui￾ties, these results suggest the applicability of our approach to other imaging modalities. Depth of trees. Fig. 6(b) shows how the depth of trees af￾fects test accuracy using either 15k or 900k images. Of all the training parameters, depth appears to have the most sig￾nificant effect as it directly impacts the model capacity of the classifier. Using only 15k images we observe overfitting beginning around depth 17, but the enlarged 900k training set avoids this. The high accuracy gradient at depth 20 sug￾gests even better results can be achieved by training still deeper trees, at a small extra run-time computational cost and a large extra memory penalty. Of practical interest is that, until about depth 10, the training set size matters little, suggesting an efficient training strategy. Maximum probe offset. The range of depth probe offsets allowed during training has a large effect on accuracy. We show this in Fig. 6(c) for 5k training images, where ‘maxi￾mum probe offset’ means the max. absolute value proposed for both x and y coordinates of u and v in Eq. 1. The con￾centric boxes on the right show the 5 tested maximum off-

1.0 bustly infer 3D body joint positions and cope naturally with cropping and translation.The speed of nearest neighbor chamfer matching is also drastically slower(2 fps)than our algorithm.While hierarchical matching [14]is faster,one Joint prediction from ground truth body parts would still need a massive exemplar set to achieve compa- Joint prediction from inferred body parts rable accuracy. 0.3 Comparison with [13].The authors of [13]provided their test data and results for direct comparison.Their algorithm uses body part proposals from [28]and further tracks the skeleton with kinematic and temporal information.Their Figure 7.Joint prediction accuracy.We compare the actual per- data comes from a time-of-flight depth camera with very formance of our system (red)with the best achievable result(blue) different noise characteristics to our structured light sen- given the ground truth body part labels. sor.Without any changes to our training data or algorithm, sets calibrated for a left shoulder pixel in that image;the Fig.8(b)shows considerably improved joint prediction av- largest offset covers almost all the body.(Recall that this erage precision.Our algorithm also runs at least 10x faster. maximum offset scales with world depth of the pixel).As Full rotations and multiple people.To evaluate the full the maximum probe offset is increased,the classifier is able 360 rotation scenario,we trained a forest on 900k images to use more spatial context to make its decisions,though containing full rotations and tested on 5k synthetic full ro- without enough data would eventually risk overfitting to this tation images (with held out poses).Despite the massive context.Accuracy increases with the maximum probe off- increase in left-right ambiguity,our system was still able set,though levels off around 129 pixel meters to achieve an mAP of 0.655,indicating that our classifier 4.3.Joint prediction accuracy can accurately learn the subtle visual cues that distinguish front and back facing poses.Residual left-right uncertainty In Fig.7 we show average precision results on the syn- thetic test set,achieving 0.731 mAP.We compare an ide- after classification can naturally be propagated to a track- ing algorithm through multiple hypotheses.Our approach alized setup that is given the ground truth body part labels can propose joint positions for multiple people in the image, to the real setup using inferred body parts.While we do since the per-pixel classifier generalizes well even without pay a small penalty for using our intermediate body parts explicit training for this scenario.Results are given in Fig.I representation,for many joints the inferred results are both and the supplementary material. highly accurate and close to this upper bound.On the real Faster proposals.We also implemented a faster alterna- test set,we have ground truth labels for head,shoulders,el- tive approach to generating the proposals based on simple bows.and hands.An mAP of 0.984 is achieved on those bottom-up clustering.Combined with body part classifica- parts given the ground truth body part labels,while 0.914 tion,this runs at~200 fps on the Xbox GPU,vs.~50 fps mAP is achieved using the inferred body parts.As expected, using mean shift on a modern 8 core desktop CPU.Given these numbers are considerably higher on this easier test set. the computational savings,the 0.677 mAP achieved on the Comparison with nearest neighbor.To highlight the need to treat pose recognition in parts,and to calibrate the dif- synthetic test set compares favorably to the 0.731 mAP of ficulty of our test set for the reader,we compare with the mean shift approach. two variants of exact nearest-neighbor whole-body match- 5.Discussion ing in Fig.8(a).The first,idealized,variant matches the We have seen how accurate proposals for the 3D loca- ground truth test skeleton to a set of training exemplar skele- tions of body joints can be estimated in super real-time from tons with optimal rigid translational alignment in 3D world single depth images.We introduced body part recognition space.Of course,in practice one has no access to the test as an intermediate representation for human pose estima- skeleton.As an example of a realizable system,the second tion.Using a highly varied synthetic training set allowed variant uses chamfer matching [14]to compare the test im- us to train very deep decision forests using simple depth- age to the training exemplars.This is computed using depth invariant features without overfitting,learning invariance to edges and 12 orientation bins.To make the chamfer task both pose and shape.Detecting modes in a density function easier,we throw out any cropped training or test images. gives the final set of confidence-weighted 3D joint propos- We align images using the 3D center of mass,and found als.Our results show high correlation between real and syn- that further local rigid translation only reduced accuracy. thetic data.and between the intermediate classification and Our algorithm,recognizing in parts,generalizes better the final joint proposal accuracy.We have highlighted the than even the idealized skeleton matching until about 150k importance of breaking the whole skeleton into parts,and training images are reached.As noted above,our results show state of the art accuracy on a competitive test set. may get even better with deeper trees,but already we ro- As future work,we plan further study of the variability

Joint prediction accuracy 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Center Head Center Neck Left Shoulder Right Shoulder Left Elbow Right Elbow Left Wrist Right Wrist Left Hand Right Hand Left Knee Right Knee Left Ankle Right Ankle Left Foot Right Foot Mean AP Average precision Joint prediction from ground truth body parts Joint prediction from inferred body parts 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Head Neck L. Shoulder R. Shoulder L. Elbow R. Elbow L. Wrist R. Wrist L. Hand R. Hand L. Knee R. Knee L. Ankle R. Ankle L. Foot R. Foot Mean AP Average precision Joint prediction from ground truth body parts Joint prediction from inferred body parts Figure 7. Joint prediction accuracy. We compare the actual per￾formance of our system (red) with the best achievable result (blue) given the ground truth body part labels. sets calibrated for a left shoulder pixel in that image; the largest offset covers almost all the body. (Recall that this maximum offset scales with world depth of the pixel). As the maximum probe offset is increased, the classifier is able to use more spatial context to make its decisions, though without enough data would eventually risk overfitting to this context. Accuracy increases with the maximum probe off￾set, though levels off around 129 pixel meters. 4.3. Joint prediction accuracy In Fig. 7 we show average precision results on the syn￾thetic test set, achieving 0.731 mAP. We compare an ide￾alized setup that is given the ground truth body part labels to the real setup using inferred body parts. While we do pay a small penalty for using our intermediate body parts representation, for many joints the inferred results are both highly accurate and close to this upper bound. On the real test set, we have ground truth labels for head, shoulders, el￾bows, and hands. An mAP of 0.984 is achieved on those parts given the ground truth body part labels, while 0.914 mAP is achieved using the inferred body parts. As expected, these numbers are considerably higher on this easier test set. Comparison with nearest neighbor. To highlight the need to treat pose recognition in parts, and to calibrate the dif- ficulty of our test set for the reader, we compare with two variants of exact nearest-neighbor whole-body match￾ing in Fig. 8(a). The first, idealized, variant matches the ground truth test skeleton to a set of training exemplar skele￾tons with optimal rigid translational alignment in 3D world space. Of course, in practice one has no access to the test skeleton. As an example of a realizable system, the second variant uses chamfer matching [14] to compare the test im￾age to the training exemplars. This is computed using depth edges and 12 orientation bins. To make the chamfer task easier, we throw out any cropped training or test images. We align images using the 3D center of mass, and found that further local rigid translation only reduced accuracy. Our algorithm, recognizing in parts, generalizes better than even the idealized skeleton matching until about 150k training images are reached. As noted above, our results may get even better with deeper trees, but already we ro￾bustly infer 3D body joint positions and cope naturally with cropping and translation. The speed of nearest neighbor chamfer matching is also drastically slower (2 fps) than our algorithm. While hierarchical matching [14] is faster, one would still need a massive exemplar set to achieve compa￾rable accuracy. Comparison with [13]. The authors of [13] provided their test data and results for direct comparison. Their algorithm uses body part proposals from [28] and further tracks the skeleton with kinematic and temporal information. Their data comes from a time-of-flight depth camera with very different noise characteristics to our structured light sen￾sor. Without any changes to our training data or algorithm, Fig. 8(b) shows considerably improved joint prediction av￾erage precision. Our algorithm also runs at least 10x faster. Full rotations and multiple people. To evaluate the full 360◦ rotation scenario, we trained a forest on 900k images containing full rotations and tested on 5k synthetic full ro￾tation images (with held out poses). Despite the massive increase in left-right ambiguity, our system was still able to achieve an mAP of 0.655, indicating that our classifier can accurately learn the subtle visual cues that distinguish front and back facing poses. Residual left-right uncertainty after classification can naturally be propagated to a track￾ing algorithm through multiple hypotheses. Our approach can propose joint positions for multiple people in the image, since the per-pixel classifier generalizes well even without explicit training for this scenario. Results are given in Fig. 1 and the supplementary material. Faster proposals. We also implemented a faster alterna￾tive approach to generating the proposals based on simple bottom-up clustering. Combined with body part classifica￾tion, this runs at ∼ 200 fps on the Xbox GPU, vs. ∼ 50 fps using mean shift on a modern 8 core desktop CPU. Given the computational savings, the 0.677 mAP achieved on the synthetic test set compares favorably to the 0.731 mAP of the mean shift approach. 5. Discussion We have seen how accurate proposals for the 3D loca￾tions of body joints can be estimated in super real-time from single depth images. We introduced body part recognition as an intermediate representation for human pose estima￾tion. Using a highly varied synthetic training set allowed us to train very deep decision forests using simple depth￾invariant features without overfitting, learning invariance to both pose and shape. Detecting modes in a density function gives the final set of confidence-weighted 3D joint propos￾als. Our results show high correlation between real and syn￾thetic data, and between the intermediate classification and the final joint proposal accuracy. We have highlighted the importance of breaking the whole skeleton into parts, and show state of the art accuracy on a competitive test set. As future work, we plan further study of the variability

08 1.0 0 -Ground truth skeleton NN 0 -Chamfer NN 50.9 05 Our algorithm 10.8 0.4 00.7 03 20.6 Our result Ganapathi et al.(tracking 0.5 0.01 (a) 30 300 3000 30000 Number of training images(log scale) 30oo00(b） Figure 8.Comparisons.(a)Comparison with nearest neighbor matching.(b)Comparison with [13].Even without the kinematic and temporal constraints exploited by [13],our algorithm is able to more accurately localize body joints. in the source mocap data,the properties of the generative [19]S.Knoop.S.Vacek,and R.Dillmann.Sensor fusion for 3D human model underlying the synthesis pipeline,and the particular body tracking with an articulated 3D body model.In Proc./CRA. part definitions.Whether a similarly efficient approach that 2006.1,2 [20]V.Lepetit,P.Lagger,and P.Fua.Randomized trees for real-time can directly regress joint positions is also an open question. keypoint recognition.In Proc.CVPR,pages 2:775-781.2005.4 Perhaps a global estimate of latent variables such as coarse [21]Microsoft Corp.Redmond WA.Kinect for Xbox 360.1,2 person orientation could be used to condition the body part [22]T.Moeslund,A.Hilton,and V.Kruiger.A survey of advances in vision-based human motion capture and analysis.CV/U.2006.2 inference and remove ambiguities in local pose estimates. (23]F.Moosmann,B.Triggs,and F.Jurie.Fast discriminative visual Acknowledgements.We thank the many skilled engineers in codebooks using randomized clustering forests.In NIPS,2006.4 [24]G.Mori and J.Malik.Estimating human body configurations using Xbox,particularly Robert Craig,Matt Bronder,Craig Peeper,Momin Al- shape context matching.In Proc./CCV,2003.2 Ghosien,and Ryan Geiss,who built the Kinect tracking system on top [25]R.Navaratnam,A.W.Fitzgibbon,and R.Cipolla.The joint manifold of this research.We also thank John Winn,Duncan Robertson,Antonio model for semi-supervised multi-valued regression.In Proc.ICCV, Criminisi,Shahram Izadi,Ollie Williams,and Mihai Budiu for help and 2007.2 valuable discussions,and Varun Ganapathi and Christian Plagemann for [26]H.Ning,W.Xu,Y.Gong,and T.S.Huang.Discriminative learning providing their test data. of visual words for 3D human pose estimation.In Proc.CVPR,2008. References (27]R.Okada and S.Soatto.Relevant feature selection for human pose estimation and localization in cluttered images.In Proc.ECCV, (1]A.Agarwal and B.Triggs.3D human pose from silhouettes by rele- 2008.2 vance vector regression.In Proc.CVPR,2004.2 [28]C.Plagemann,V.Ganapathi,D.Koller,and S.Thrun.Real-time [2]Y.Amit and D.Geman.Shape quantization and recognition with identification and localization of body parts from depth images.In randomized trees.Neural Computation,9(7):1545-1588,1997.4 Poc.1CRA.2010.1.2.7 [3]D.Anguelov,B.Taskar,V.Chatalbashev,D.Koller,D.Gupta,and [29]R.Poppe.Vision-based human motion analysis:An overview.CVIU, A.Ng.Discriminative learning of markov random fields for segmen- 108.2007.2 tation of 3D scan data.In Proc.CVPR,2005.2 [30]J.R.Quinlan.Induction of decision trees.Mach.Learn,1986.4 [4]Autodesk MotionBuilder.3 [31]D.Ramanan and D.Forsyth.Finding and tracking people from the [5]S.Belongie,J.Malik,and J.Puzicha.Shape matching and object bottom up.In Proc.CVPR,2003.2 recognition using shape contexts.IEEE Trans.PAM/.24.2002.4 [32]G.Rogez,J.Rihan,S.Ramalingam,C.Orrite,and P.Torr.Random- [6]L.Bourdev and J.Malik.Poselets:Body part detectors trained using ized trees for human pose detection.In Proc.CVPR,2008.2 3D human pose annotations.In Proc.ICCV,2009.2 [33]G.Shakhnarovich,P.Viola,and T.Darrell.Fast pose estimation with (7]C.Bregler and J.Malik.Tracking people with twists and exponential parameter sensitive hashing.In Proc./CCV,2003.2 maps.In Proc.CVPR,1998.1.2 34]T.Sharp.Implementing decision trees and forests on a GPU.In Proc ECCV,2008.1,4 [8]L.Breiman.Random forests.Mach.Learning,45(1):5-32.2001.4 [35]B.Shepherd.An appraisal of a decision tree approach to image clas [9]CMU Mocap Database.http://mocap.cs.cmu.edu/.3 sification.In /CA/.1983.4 [10]D.Comaniciu and P.Meer.Mean shift:A robust approach toward [36]J.Shotton,M.Johnson,and R.Cipolla.Semantic texton forests for feature space analysis.IEEE Trans.PAMI,24(5),2002.1,5 image categorization and segmentation.In Proc.CVPR,2008.4 [11]P.Felzenszwalb and D.Huttenlocher.Pictorial structures for object [37]M.Siddiqui and G.Medioni.Human pose estimation from a single recognition.I/CV,61(1):55-79,Jan.2005.2 view point,real-time range sensor.In CVCG at CVPR,2010.1.2 [12]R.Fergus,P.Perona,and A.Zisserman.Object class recognition by [38]H.Sidenbladh,M.Black,and L.Sigal.Implicit probabilistic models unsupervised scale-invariant learning.In Proc.CVPR.2003.1 of human motion for synthesis and tracking.In ECCV,2002.2 [13]V.Ganapathi.C.Plagemann,D.Koller,and S.Thrun.Real time [39]L.Sigal,S.Bhatia,S.Roth,M.Black.and M.Isard.Tracking loose- motion capture using a single time-of-flight camera.In Proc.CVPR. limbed people.In Proc.CVPR,2004.1,2 2010.1.5.7.8 [40]Z.Tu.Auto-context and its application to high-level vision tasks.In [14]D.Gavrila.Pedestrian detection from a moving vehicle.In Proc. Proc.CVPR.2008.2 ECCV.June 2000.7 (41]R.Urtasun and T.Darrell.Local probabilistic regression for activity. [15]T.Gonzalez.Clustering to minimize the maximum intercluster dis- independent human pose inference.In Proc.CVPR,2008.2 tance.Theor.Comp.Sci.,38,1985.3 [42]R.Wang and J.Popovic.Real-time hand-tracking with a color glove. [16]D.Grest.J.Woetzel,and R.Koch.Nonlinear body pose estimation In Proc.ACM SIGGRAPH,2009.1.2 from depth images.InIn Proc.DAGM,2005.1,2 (43]J.Winn and J.Shotton.The layout consistent random field for rec- [17]S.Ioffe and D.Forsyth.Probabilistic methods for finding people ognizing and segmenting partially occluded objects.In Proc.CVPR. JCV.43(1):45-68.2001.2 2006.1 [18]E.Kalogerakis,A.Hertzmann,and K.Singh.Learning 3D mesh [44]Y.Zhu and K.Fujimura.Constrained optimization for human pose segmentation and labeling.ACM Trans.Graphics.29(3),2010.2 estimation from depth sequences.In Proc.ACCV,2007.1,2

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 30 300 3000 30000 300000 Mean average precision Number of training images (log scale) Ground truth skeleton NN Chamfer NN Our algorithm 0.5 0.6 0.7 0.8 0.9 1.0 Head Neck L. Shoulder R. Shoulder L. Elbow R. Elbow L. Wrist R. Wrist L. Hand R. Hand L. Knee R. Knee L. Ankle R. Ankle L. Foot R. Foot Mean AP Average precision Our result (per frame) Ganapathi et al. (tracking) Combined Comparisons (a) (b) Figure 8. Comparisons. (a) Comparison with nearest neighbor matching. (b) Comparison with [13]. Even without the kinematic and temporal constraints exploited by [13], our algorithm is able to more accurately localize body joints. in the source mocap data, the properties of the generative model underlying the synthesis pipeline, and the particular part definitions. Whether a similarly efficient approach that can directly regress joint positions is also an open question. Perhaps a global estimate of latent variables such as coarse person orientation could be used to condition the body part inference and remove ambiguities in local pose estimates. Acknowledgements. We thank the many skilled engineers in Xbox, particularly Robert Craig, Matt Bronder, Craig Peeper, Momin Al￾Ghosien, and Ryan Geiss, who built the Kinect tracking system on top of this research. We also thank John Winn, Duncan Robertson, Antonio Criminisi, Shahram Izadi, Ollie Williams, and Mihai Budiu for help and valuable discussions, and Varun Ganapathi and Christian Plagemann for providing their test data. References [1] A. Agarwal and B. Triggs. 3D human pose from silhouettes by rele￾vance vector regression. In Proc. CVPR, 2004. 2 [2] Y. Amit and D. Geman. Shape quantization and recognition with randomized trees. Neural Computation, 9(7):1545–1588, 1997. 4 [3] D. Anguelov, B. Taskar, V. Chatalbashev, D. Koller, D. Gupta, and A. Ng. Discriminative learning of markov random fields for segmen￾tation of 3D scan data. In Proc. CVPR, 2005. 2 [4] Autodesk MotionBuilder. 3 [5] S. Belongie, J. Malik, and J. Puzicha. Shape matching and object recognition using shape contexts. IEEE Trans. PAMI, 24, 2002. 4 [6] L. Bourdev and J. Malik. Poselets: Body part detectors trained using 3D human pose annotations. In Proc. ICCV, 2009. 2 [7] C. Bregler and J. Malik. Tracking people with twists and exponential maps. In Proc. CVPR, 1998. 1, 2 [8] L. Breiman. Random forests. Mach. Learning, 45(1):5–32, 2001. 4 [9] CMU Mocap Database. http://mocap.cs.cmu.edu/. 3 [10] D. Comaniciu and P. Meer. Mean shift: A robust approach toward feature space analysis. IEEE Trans. PAMI, 24(5), 2002. 1, 5 [11] P. Felzenszwalb and D. Huttenlocher. Pictorial structures for object recognition. IJCV, 61(1):55–79, Jan. 2005. 2 [12] R. Fergus, P. Perona, and A. Zisserman. Object class recognition by unsupervised scale-invariant learning. In Proc. CVPR, 2003. 1 [13] V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun. Real time motion capture using a single time-of-flight camera. In Proc. CVPR, 2010. 1, 5, 7, 8 [14] D. Gavrila. Pedestrian detection from a moving vehicle. In Proc. ECCV, June 2000. 7 [15] T. Gonzalez. Clustering to minimize the maximum intercluster dis￾tance. Theor. Comp. Sci., 38, 1985. 3 [16] D. Grest, J. Woetzel, and R. Koch. Nonlinear body pose estimation from depth images. In In Proc. DAGM, 2005. 1, 2 [17] S. Ioffe and D. Forsyth. Probabilistic methods for finding people. IJCV, 43(1):45–68, 2001. 2 [18] E. Kalogerakis, A. Hertzmann, and K. Singh. Learning 3D mesh segmentation and labeling. ACM Trans. Graphics, 29(3), 2010. 2 [19] S. Knoop, S. Vacek, and R. Dillmann. Sensor fusion for 3D human body tracking with an articulated 3D body model. In Proc. ICRA, 2006. 1, 2 [20] V. Lepetit, P. Lagger, and P. Fua. Randomized trees for real-time keypoint recognition. In Proc. CVPR, pages 2:775–781, 2005. 4 [21] Microsoft Corp. Redmond WA. Kinect for Xbox 360. 1, 2 [22] T. Moeslund, A. Hilton, and V. Kruger. A survey of advances in ¨ vision-based human motion capture and analysis. CVIU, 2006. 2 [23] F. Moosmann, B. Triggs, and F. Jurie. Fast discriminative visual codebooks using randomized clustering forests. In NIPS, 2006. 4 [24] G. Mori and J. Malik. Estimating human body configurations using shape context matching. In Proc. ICCV, 2003. 2 [25] R. Navaratnam, A. W. Fitzgibbon, and R. Cipolla. The joint manifold model for semi-supervised multi-valued regression. In Proc. ICCV, 2007. 2 [26] H. Ning, W. Xu, Y. Gong, and T. S. Huang. Discriminative learning of visual words for 3D human pose estimation. In Proc. CVPR, 2008. 2 [27] R. Okada and S. Soatto. Relevant feature selection for human pose estimation and localization in cluttered images. In Proc. ECCV, 2008. 2 [28] C. Plagemann, V. Ganapathi, D. Koller, and S. Thrun. Real-time identification and localization of body parts from depth images. In Proc. ICRA, 2010. 1, 2, 7 [29] R. Poppe. Vision-based human motion analysis: An overview. CVIU, 108, 2007. 2 [30] J. R. Quinlan. Induction of decision trees. Mach. Learn, 1986. 4 [31] D. Ramanan and D. Forsyth. Finding and tracking people from the bottom up. In Proc. CVPR, 2003. 2 [32] G. Rogez, J. Rihan, S. Ramalingam, C. Orrite, and P. Torr. Random￾ized trees for human pose detection. In Proc. CVPR, 2008. 2 [33] G. Shakhnarovich, P. Viola, and T. Darrell. Fast pose estimation with parameter sensitive hashing. In Proc. ICCV, 2003. 2 [34] T. Sharp. Implementing decision trees and forests on a GPU. In Proc. ECCV, 2008. 1, 4 [35] B. Shepherd. An appraisal of a decision tree approach to image clas￾sification. In IJCAI, 1983. 4 [36] J. Shotton, M. Johnson, and R. Cipolla. Semantic texton forests for image categorization and segmentation. In Proc. CVPR, 2008. 4 [37] M. Siddiqui and G. Medioni. Human pose estimation from a single view point, real-time range sensor. In CVCG at CVPR, 2010. 1, 2 [38] H. Sidenbladh, M. Black, and L. Sigal. Implicit probabilistic models of human motion for synthesis and tracking. In ECCV, 2002. 2 [39] L. Sigal, S. Bhatia, S. Roth, M. Black, and M. Isard. Tracking loose￾limbed people. In Proc. CVPR, 2004. 1, 2 [40] Z. Tu. Auto-context and its application to high-level vision tasks. In Proc. CVPR, 2008. 2 [41] R. Urtasun and T. Darrell. Local probabilistic regression for activity￾independent human pose inference. In Proc. CVPR, 2008. 2 [42] R. Wang and J. Popovic. Real-time hand-tracking with a color glove. ´ In Proc. ACM SIGGRAPH, 2009. 1, 2 [43] J. Winn and J. Shotton. The layout consistent random field for rec￾ognizing and segmenting partially occluded objects. In Proc. CVPR, 2006. 1 [44] Y. Zhu and K. Fujimura. Constrained optimization for human pose estimation from depth sequences. In Proc. ACCV, 2007. 1, 2