使用3DCNN和卷积LSTM进行手势识别学习时空特征

研究论文

195 浏览量 2021-02-25 07:31:22 上传评论 7 收藏 690KB PDF 举报

资源推荐

资源详情

资源评论

Learning Spatiotemporal Features using 3DCNN and Convolutional LSTM for

Gesture Recognition

Liang Zhang, Guangming Zhu, Peiyi Shen, Juan Song

School of Software, Xidian University

{liangzhang, gmzhu, pyshen, songjuan}@xidian.edu.cn

Syed Afaq Shah, Mohammed Bennamoun

University of Western Australia

{afaq.shah, mohammed.bennamoun}@uwa.edu.au

Abstract

Gesture recognition aims at understanding the ongoing

human gestures. In this paper, we present a deep archi-

tecture to learn spatiotemporal features for gesture recog-

nition. The deep architecture ﬁrst learns 2D spatiotempo-

ral feature maps using 3D convolutional neural networks

(3DCNN) and bidirectional convolutional long-short-term-

memory networks (ConvLSTM). The learnt 2D feature maps

can encode the global temporal information and local spa-

tial information simultaneously. Then, 2DCNN is utilized

further to learn the higher-level spatiotemporal features

from the 2D feature maps for the ﬁnal gesture recogni-

tion. The spatiotemporal correlation information is kept

through the whole process of feature learning. This makes

the deep architecture an effective spatiotemporal feature

learner. Experiments on the ChaLearn LAP large-scale iso-

lated gesture dataset (IsoGD) and the Shefﬁeld Kinect Ges-

ture (SKIG) dataset demonstrate the superiority of the pro-

posed deep architecture.

1. Introduction

Gestures, as a nonverbal body language, play a very

important role in humans daily life. Gesture recognition

aims at understanding the ongoing human gestures and is

of great signiﬁcance for human-robot/computer interaction,

sign language recognition and virtual [23].

Effective and universal gesture recognition from videos

is extremely difﬁcult; partly due to the large gesture vocab-

ularies with cultural differences, various illumination con-

ditions, out-of-vocabulary motions, inconsistent and non-

standard behaviors among different performers, etc [12].

Moreover, gestures have various time durations and involve

different body parts. A small handful of gestures can be

Figure 1. Overview of the proposed deep architecture. 3DCNN

and bidirectional ConvLSTM are utilized to learn the short-

term and long-term spatiotemporal features successively, and then

2DCNN is used to learn higher-level spatiotemporal features based

on the learnt 2D long-term spatiotemporal feature maps for the ﬁ-

nal gesture recognition.

represented by a single posture of hands and arms, but most

of the gestures are composed of a sequence of hand and arm

postures. Therefore, learning effective spatiotemporal fea-

tures is crucially important for robust gesture recognition.

According to [32], there are four typical properties for ef-

fective spatiotemporal features of gestures: (i) generic, (ii)

compact, (iii) efﬁcient to compute, and (iv) simple to imple-

ment.

Inspired by the deep learning breakthroughs in image

recognition [17, 29, 31], lots of neural network based

frameworks are proposed to learn spatiotemporal features

3120

Figure 2. Pipeline of the proposed framework. Multimodal data

is used to train the proposed deep architecture respectively, and

RGB/Depth/Flow based spatiotemporal features are extracted and

combined into large multimodal spatiotemporal feature vectors

further. Linear SVM classiﬁer is utilized for the ﬁnal gesture

recognition.

for human action/gesture recognition. Two-Stream Con-

volutional Networks [28] learn spatial and temporal fea-

tures separately. Long-term Recurrent Convolutional Net-

works (LRCN) [6] learn spatial and temporal features us-

ing convolutional neural networks (CNN) and long-short-

term-memory (LSTM) networks successively. Tran et

al. [32] constructed a deep 3D ConvNet to learn spatiotem-

poral features directly and achieved the best performance

on different types of video analysis tasks. Molchanov et

al. [24] proposed to ﬁrst learn spatiotemporal features on

each clip using 3DCNN, and then to fuse the spatiotem-

poral features over the whole video using recurrent neu-

ral networks (RNN). Obviously, 3DCNN is superior to

learn spatiotemporal features for gesture recognition. How-

ever, RNN/LSTM based networks are more suitable to en-

code long-term temporal information, especially from the

various-length videos. Although Molchanov et al. [24] pro-

posed to combine 3DCNN and RNN, fully-connected spa-

tiotemporal features are transferred into RNN, this make the

spatial correlation information lost in the RNN stage.

In this paper, we propose to ﬁrst learn short-term spa-

tiotemporal features using a shallow 3DCNN, and then

learn long-term spatiotemporal features further using bidi-

rectional convolutional LSTM (ConvLSTM), lastly recog-

nize gestures using 2DCNN based on the learnt 2D spa-

tiotemporal feature maps. An overview of the proposed

deep architecture is illustrated in Figure 1, and the pipeline

of the proposed framework is in Figure 2.

In brief, our contributions in this paper include:

• 2D spatiotemporal feature maps are learnt using

3DCNN and bidirectional convolutional LSTM. The

2D feature maps can encode the global temporal infor-

mation and local spatial information. Spatiotemporal

correlation information is kept through the whole fea-

ture map learning process.

• The proposed deep architecture can transform video

ﬁles into 2D spatiotemporal feature maps. This trans-

formation makes the deep architecture more extensi-

ble to utilize the state-of-the-art 2DCNN to learn the

higher-level spatiotemporal features for gesture recog-

nition.

• The proposed spatiotemporal features with a linear

SVM classiﬁcation model outperform or achieve per-

formance in par with the state-of-the-art methods on

two different benchmarks.

• To the best of our knowledge, this is the ﬁrst time to

learn 2D spatiotemporal feature maps using 3DCNN

and bidirectional ConvLSTM, and then to learn higher-

level spatiotemporal features using 2DCNN for the ﬁ-

nal gesture recognition.

2. Related Work

Learning spatiotemporal features is crucial for effective

human action/gesture recognition. Various deep neural net-

works have been proposed recently [15]. However, gesture

recognition has signiﬁcant differences from action recog-

nition. One obvious difference is that backgrounds may

be an effective clue for action recognition, but in con-

trast can be a challenging factor for gesture recognition.

For example, scene backgrounds can help recognize hu-

man actions, especially the sports in UCF101 [30], but

they may bring negative impact on gesture recognition per-

formance. In fact, gestures focus more on the movement

of hands and arms. Thus, two-stream ConvNets [28] and

their derivations [36, 13] obtain the state-of-the-art perfor-

mance on HMDB51 [18] and UCF101 datasets, but they

fail to achieve a similar performance in the case of ges-

ture recognition. Another obvious approach is to learn spa-

tial and temporal features successively, such as LRCN [6].

However, Pigou et al. [26] demonstrated that LRCN-style

networks are not optimal, while bidirectional recurrence

and temporal convolutions can improve gesture recognition

performance signiﬁcantly. The huge success of 2DCNN

on image recognition has encouraged researchers to trans-

form video ﬁles into particular 2D image ﬁles, so that the

state-of-the-art 2DCNN networks can be applied on gesture

recognition [39]. But, handcrafted transformation methods

have inherent deﬁciency on adaptive learning. In this paper,

a deep architecture will be described, which can learn adap-

tively to transform gesture video ﬁles into 2D spatiotempo-

ral feature maps.

Tran et al. [32] constructed a deep 3D ConvNet to learn

spatiotemporal features directly and achieved the best per-

formance on different types of video analysis tasks. In-

spired by [32], 3DCNN-based neural networks obtained the

remarkable performances on gesture recognition [11]. In

the past 2016 ChaLearn LAP Large-scale Isolated/ Continu-

ous Gesture Recognition Challenges [35], 3DCNN demon-

3121

剩余8页未读，继续阅读

评论收藏

内容反馈

weixin_38608866

粉丝: 7

使用3DCNN和卷积LSTM进行手势识别学习时空特征

基于CNN卷积神经网络的手势识别系统.pdf

基于卷积神经网络的手势识别

基于三维卷积和卷积LSTM的多模态手势识别

TensorFlow入门：使用卷积网络模型实现手势识别

用于3D动作识别和检测的基于空时注意力对的LSTM网络

Neuro-captioner：通过从CNN中提取特征，然后从LSTM中构造一个句子来为给定图像加标题

3D CNN 手势识别moviNet_tf2.0-python源码.zip

Gesture-recognition-using-CNNLSTM:使用GRIT数据集，构建了将2D CNN与LSTM相结合的模型，以从webCam视频提要中执行实时手势识别。 也使用LSTM建立了使用3D CNN的另一个模型

CNN手势识别

TS-LSTM:使用TS-LSTM模型的基于骨架的动作识别

GestureRecognition-PyTorch:动作识别网络-CNN + LSTM

手势识别算法总结

GestureRecognition:使用TensorFlow进行手势识别

3D-CNN-3D-images-Tensorflow.zip

利用tensorflow实现3DCNN

图像字幕生成器：LSTM模型从预先训练的VGG-16模型中提取特征后，会为输入图像生成字幕。 （计算机视觉，自然语言处理，深度学习，Python）

3D 卷积神经网络：将任意 2D CNN 扩展为 3D CNN，扩展了 Mathworks 大部分官方预训练权重模型的通用性！-matlab开发

3DCNN一篇经典论文

Python-手势识别使用在TensorFlow中卷积神经网络实现

CNN-3D-images-Tensorflow：使用CNN（卷积神经网络）进行3D图像分类

video-classification-3d-cnn-pytorch：使用3D ResNet的视频分类工具

3D-ResNets-PyTorch：用于动作识别的3D ResNets（CVPR 2018）

3D_CNN_tensorflow：用于车辆检测的KITTI数据处理和3D CNN

3D ResNet：用于动作识别的3D ResNet

3D-ResNets-PyTorch-master.zip

基于改进3D-CNN的多源遥感数据树种识别

3D unet CNN网络

CNN-LSTM混合模型用于台风形成预报

最新资源

Gesture-recognition-using-CNNLSTM:使用GRIT数据集，构建了将2D CNN与LSTM相结合的模型，以从webCam视频提要中执行实时手势识别。也使用LSTM建立了使用3D CNN的另一个模型

图像字幕生成器：LSTM模型从预先训练的VGG-16模型中提取特征后，会为输入图像生成字幕。（计算机视觉，自然语言处理，深度学习，Python）