Stochastic Systems Group
Home Research Group Members Programs  
Demos Calendar Publications Mission Statement Alumni

SSG Seminar Abstract

Visual Hand Tracking Using Nonparametric Belief Propagation

Erik Sudderth - SSG, MIT

Accurate visual detection and tracking of three-dimensional articulated objects is a challenging problem with applications in human-computer interfaces, motion capture, and scene understanding. In this talk, we describe a probabilistic method for tracking a geometric hand model from monocular image sequences. We first show that the kinematic constraints implied by the model's joints have a probabilistic structure which is well described by a graphical model. Inference in this model is complicated by the hand's many degrees of freedom, as well as multimodal likelihoods caused by ambiguous image measurements. We use nonparametric belief propagation (NBP) to develop a tracking algorithm which exploits the graph's structure to control complexity, while avoiding costly discretization.

While kinematic constraints naturally have a local structure, self-occlusions created by the imaging process lead to complex interpendencies in color and edge-based likelihood functions. However, we show that local structure may be recovered by introducing binary hidden variables describing the occlusion state of each pixel. We augment the NBP algorithm to infer these occlusion variables in a distributed fashion, producing hand position estimates which properly account for occlusion events. We provide results demonstrating that NBP may be used to refine inaccurate model initializations, as well as track hand motion through extended image sequences.

Problems with this site should be emailed to