Who is where? Matching people in video to wearable acceleration during crowded mingling events

Abstract

We address the challenging problem of associating acceler- ation data from a wearable sensor with the corresponding spatio-temporal region of a person in video during crowded mingling scenarios. This is an important rst step for multi- sensor behavior analysis using these two modalities. Clearly, as the numbers of people in a scene increases, there is also a need to robustly and automatically associate a region of the video with each person's device. We propose a hierarchi- cal association approach which exploits the spatial context of the scene, outperforming the state-of-the-art approaches signi cantly. Moreover, we present experiments on match- ing from 3 to more than 130 acceleration and video streams which, to our knowledge, is signi cantly larger than prior works where only up to 5 device streams are associated.