Deep Bilinear Learning for RGB-D Action Recognition

Jian-Fang  Hu; Wei-Shi Zheng; Jiahui  Pan; Jianhuang Lai; Jianguo Zhang

doi:10.1007/978-3-030-01234-2_21

Deep Bilinear Learning for RGB-D Action Recognition

Jian-Fang Hu, Wei-Shi Zheng (Lead / Corresponding author), Jiahui Pan, Jianhuang Lai, Jianguo Zhang

Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

23 Citations (Scopus)

Abstract

In this paper, we focus on exploring modality-temporal mutual information for RGB-D action recognition. In order to learn time-varying information and multi-modal features jointly, we propose a novel deep bilinear learning framework. In the framework, we propose bilinear blocks that consist of two linear pooling layers for pooling the input cube features from both modality and temporal directions, separately. To capture rich modality-temporal information and facilitate our deep bilinear learning, a new action feature called modality-temporal cube is presented in a tensor structure for characterizing RGB-D actions from a comprehensive perspective. Our method is extensively tested on two public datasets with four different evaluation settings, and the results show that the proposed method outperforms the state-of-the-art approaches.

Original language	English
Title of host publication	ECCV 2018
Subtitle of host publication	Computer Vision - ECCV 2018
Editors	Vittorio Ferrari, Cristian Sminchisescu, Martial Hebert, Yair Weiss
Place of Publication	Switzerland
Publisher	Springer
Pages	346-362
Number of pages	17
Volume	11211
ISBN (Electronic)	9783030012342
ISBN (Print)	9783030012335
DOIs	https://doi.org/10.1007/978-3-030-01234-2_21
Publication status	Published - 2018
Event	European Conference on Computer Vision 2018 - Munich, Germany Duration: 8 Sept 2018 → 14 Sept 2018 https://eccv2018.org/

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11211 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Computer Vision 2018
Abbreviated title	ECCV 2018
Country/Territory	Germany
City	Munich
Period	8/09/18 → 14/09/18
Internet address	https://eccv2018.org/

Keywords

Cube
Deep bilinear
Feature learning
RGB-D action

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-030-01234-2_21

Cite this

Hu, J.-F., Zheng, W.-S., Pan, J., Lai, J., & Zhang, J. (2018). Deep Bilinear Learning for RGB-D Action Recognition. In V. Ferrari, C. Sminchisescu, M. Hebert, & Y. Weiss (Eds.), ECCV 2018: Computer Vision - ECCV 2018 (Vol. 11211, pp. 346-362). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11211 LNCS). Springer . https://doi.org/10.1007/978-3-030-01234-2_21

Hu, Jian-Fang ; Zheng, Wei-Shi ; Pan, Jiahui et al. / Deep Bilinear Learning for RGB-D Action Recognition. ECCV 2018: Computer Vision - ECCV 2018. editor / Vittorio Ferrari ; Cristian Sminchisescu ; Martial Hebert ; Yair Weiss. Vol. 11211 Switzerland : Springer , 2018. pp. 346-362 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{e1a35738b4664316940fc2552398232f,

title = "Deep Bilinear Learning for RGB-D Action Recognition",

abstract = "In this paper, we focus on exploring modality-temporal mutual information for RGB-D action recognition. In order to learn time-varying information and multi-modal features jointly, we propose a novel deep bilinear learning framework. In the framework, we propose bilinear blocks that consist of two linear pooling layers for pooling the input cube features from both modality and temporal directions, separately. To capture rich modality-temporal information and facilitate our deep bilinear learning, a new action feature called modality-temporal cube is presented in a tensor structure for characterizing RGB-D actions from a comprehensive perspective. Our method is extensively tested on two public datasets with four different evaluation settings, and the results show that the proposed method outperforms the state-of-the-art approaches.",

keywords = "Cube, Deep bilinear, Feature learning, RGB-D action",

author = "Jian-Fang Hu and Wei-Shi Zheng and Jiahui Pan and Jianhuang Lai and Jianguo Zhang",

note = "This work was supported partially by the NSFC (No.61702567, 1522115,61472456, 61661130157, 61628212).The corresponding author for this paper is Wei-Shi Zheng.; European Conference on Computer Vision 2018, ECCV 2018 ; Conference date: 08-09-2018 Through 14-09-2018",

year = "2018",

doi = "10.1007/978-3-030-01234-2_21",

language = "English",

isbn = "9783030012335",

volume = "11211",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer ",

pages = "346--362",

editor = "Vittorio Ferrari and Cristian Sminchisescu and Martial Hebert and Yair Weiss",

booktitle = "ECCV 2018",

url = "https://eccv2018.org/",

}

Hu, J-F, Zheng, W-S, Pan, J, Lai, J & Zhang, J 2018, Deep Bilinear Learning for RGB-D Action Recognition. in V Ferrari, C Sminchisescu, M Hebert & Y Weiss (eds), ECCV 2018: Computer Vision - ECCV 2018. vol. 11211, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11211 LNCS, Springer , Switzerland, pp. 346-362, European Conference on Computer Vision 2018, Munich, Germany, 8/09/18. https://doi.org/10.1007/978-3-030-01234-2_21

Deep Bilinear Learning for RGB-D Action Recognition. / Hu, Jian-Fang ; Zheng, Wei-Shi (Lead / Corresponding author); Pan, Jiahui et al.
ECCV 2018: Computer Vision - ECCV 2018. ed. / Vittorio Ferrari; Cristian Sminchisescu; Martial Hebert; Yair Weiss. Vol. 11211 Switzerland: Springer , 2018. p. 346-362 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11211 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Deep Bilinear Learning for RGB-D Action Recognition

AU - Hu, Jian-Fang

AU - Zheng, Wei-Shi

AU - Pan, Jiahui

AU - Lai, Jianhuang

AU - Zhang, Jianguo

N1 - This work was supported partially by the NSFC (No.61702567, 1522115,61472456, 61661130157, 61628212).The corresponding author for this paper is Wei-Shi Zheng.

PY - 2018

Y1 - 2018

N2 - In this paper, we focus on exploring modality-temporal mutual information for RGB-D action recognition. In order to learn time-varying information and multi-modal features jointly, we propose a novel deep bilinear learning framework. In the framework, we propose bilinear blocks that consist of two linear pooling layers for pooling the input cube features from both modality and temporal directions, separately. To capture rich modality-temporal information and facilitate our deep bilinear learning, a new action feature called modality-temporal cube is presented in a tensor structure for characterizing RGB-D actions from a comprehensive perspective. Our method is extensively tested on two public datasets with four different evaluation settings, and the results show that the proposed method outperforms the state-of-the-art approaches.

AB - In this paper, we focus on exploring modality-temporal mutual information for RGB-D action recognition. In order to learn time-varying information and multi-modal features jointly, we propose a novel deep bilinear learning framework. In the framework, we propose bilinear blocks that consist of two linear pooling layers for pooling the input cube features from both modality and temporal directions, separately. To capture rich modality-temporal information and facilitate our deep bilinear learning, a new action feature called modality-temporal cube is presented in a tensor structure for characterizing RGB-D actions from a comprehensive perspective. Our method is extensively tested on two public datasets with four different evaluation settings, and the results show that the proposed method outperforms the state-of-the-art approaches.

KW - Cube

KW - Deep bilinear

KW - Feature learning

KW - RGB-D action

UR - http://www.scopus.com/inward/record.url?scp=85055084194&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-01234-2_21

DO - 10.1007/978-3-030-01234-2_21

M3 - Conference contribution

SN - 9783030012335

VL - 11211

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 346

EP - 362

BT - ECCV 2018

A2 - Ferrari, Vittorio

A2 - Sminchisescu, Cristian

A2 - Hebert, Martial

A2 - Weiss, Yair

PB - Springer

CY - Switzerland

T2 - European Conference on Computer Vision 2018

Y2 - 8 September 2018 through 14 September 2018

ER -

Hu JF, Zheng WS, Pan J, Lai J, Zhang J. Deep Bilinear Learning for RGB-D Action Recognition. In Ferrari V, Sminchisescu C, Hebert M, Weiss Y, editors, ECCV 2018: Computer Vision - ECCV 2018. Vol. 11211. Switzerland: Springer . 2018. p. 346-362. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2018 Oct 6. doi: 10.1007/978-3-030-01234-2_21

Deep Bilinear Learning for RGB-D Action Recognition

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this