Early Action Prediction by Soft Regression

Jian-Fang Hu, Wei-Shi Zheng, Lianyang Ma, Gang Wang, Jianguo Zhang (Lead / Corresponding author)

Research output: Contribution to journalArticlepeer-review

71 Citations (Scopus)
910 Downloads (Pure)


We propose a novel approach for predicting on-going action with the assistance of a low-cost depth camera. Our approach introduces a soft regression-based early prediction framework. In this framework, we estimate soft labels for the subsequences at different progress levels, jointly learned with an action predictor. Our formulation of soft regression framework 1) overcomes a usual assumption in existing early action prediction systems that the progress level of on-going sequence is given in the testing stage; and 2) presents a theoretical framework to better resolve the ambiguity and uncertainty of subsequences at early performing stage. The proposed soft regression framework is further enhanced in order to take the relationships among subsequences and the discrepancy of soft labels over different classes into consideration, so that a Multiple Soft labels Recurrent Neural Network (MSRNN) is finally developed. For real-time performance, we also introduce "local accumulative frame feature (LAFF)", which can be computed efficiently by constructing an integral feature map. Our experiments on three RGB-D benchmark datasets and an unconstrained RGB action set demonstrate that the proposed regression-based early action prediction model outperforms existing models and the early action prediction on RGB-D sequence is more accurate than that on RGB channel.

Original languageEnglish
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number11
Early online date8 Aug 2018
Publication statusPublished - Nov 2019


  • Cameras
  • Computational modeling
  • Early action prediction
  • Feature extraction
  • Predictive models
  • RGB-D
  • Real-time systems
  • Recurrent neural networks
  • Skeleton
  • soft regression

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'Early Action Prediction by Soft Regression'. Together they form a unique fingerprint.

Cite this