Global-Local Temporal Saliency Action Prediction

Action prediction on a partially observed action sequence is a very challenging task. To address this challenge, we first design a global-local distance model, where a globaltemporal distance compares subsequences as a whole and localtemporal distance focuses on individual segment. Our distance model introduces temporal saliency for each segment to adapt its contribution. Finally, a global-local temporal action prediction model is formulated in order to jointly learn and fuse these two types of distances. Such a prediction model is capable of recognizing action of 1) an on-going sequence and 2) a sequence with arbitrarily frames missing between the beginning and end (known as gap-filling). Our proposed model is tested and compared to related action prediction models on BIT, UCF11 and HMDB datasets. The results demonstrated the effectiveness of our proposal. In particular, we showed the benefit of our proposed model on predicting unseen action types and the advantage on addressing the gapfilling problem as compared to recently developed action prediction models