Deeplearningfortimeseriesclassificationareview.pdf

自然语言处理

需积分: 50 135 浏览量 2020-03-29 10:30:26 上传评论收藏 4.79MB PDF 举报

资源推荐

资源详情

资源评论

Deep learning for time series classiﬁcation: a review

Hassan Ismail Fawaz · Germain

Forestier · Jonathan Weber · Lhassane

Idoumghar · Pierre-Alain Muller

Received: date / Accepted: date

Abstract Time Series Classiﬁcation (TSC) is an important and challenging

problem in data mining. With the increase of time series data availability, hun-

dreds of TSC algorithms have been proposed. Among these methods, only a

few have considered Deep Neural Networks (DNNs) to perform this task. This

is surprising as deep learning has seen very successful applications in the last

years. DNNs have indeed revolutionized the ﬁeld of computer vision especially

with the advent of novel deeper architectures such as Residual and Convolu-

tional Neural Networks. Apart from images, sequential data such as text and

audio can also be processed with DNNs to reach state of the art performance

for document classiﬁcation and speech recognition. In this article, we study

the current state of the art performance of deep learning algorithms for TSC

by presenting an empirical study of the most recent DNN architectures for

TSC. We give an overview of the most successful deep learning applications

in various time series domains under a uniﬁed taxonomy of DNNs for TSC.

We also provide an open source deep learning framework to the TSC commu-

nity where we implemented each of the compared approaches and evaluated

them on a univariate TSC benchmark (the UCR archive) and 12 multivariate

time series datasets. By training 8,730 deep learning models on 97 time series

datasets, we propose the most exhaustive study of DNNs for TSC to date.

Keywords deep learning · time series · classiﬁcation · review

IRIMAS, Université Haute Alsace

12 Rue des Frères Lumière,

68093 Mulhouse, France

Tel.: +33-3-89336960

E-mail: {hassan.ismail-fawaz,germain.forestier,jonathan.weber,lhassane.idoumghar,pierre-

alain.muller}@uha.fr

arXiv:1809.04356v1 [cs.LG] 12 Sep 2018

2 Hassan Ismail Fawaz et al.

1 Introduction

During the last two decades, Time Series Classiﬁcation (TSC) has been con-

sidered as one of the most challenging problems in data mining (Yang and

Wu, 2006; Esling and Agon, 2012). With the increase of temporal data avai-

lability (Silva et al., 2018), hundreds of TSC algorithms have been proposed

since 2015 (Bagnall et al., 2017). Due to their natural temporal ordering, time

series data are present in almost every task that requires some sort of human

cognitive process (Längkvist et al., 2014). In fact, any classiﬁcation problem,

using data that is registered taking into account some notion of ordering,

can be casted as a TSC problem (Cristian Borges Gamboa, 2017). Time se-

ries are encountered in many real-world applications ranging from electronic

health records (Rajkomar et al., 2018) and human activity recognition (Nweke

et al., 2018) to acoustic scene classiﬁcation (Nwe et al., 2017) and steel qual-

ity prediction (Mehdiyev et al., 2017). In addition, the diversity of the UCR

archive’s datasets’ (Chen et al., 2015b) types (the largest repository of time

series datasets) shows the diﬀerent applications of the TSC problem.

Given the need to accurately classify time series data, researchers have pro-

posed hundreds of methods to solve this task (Bagnall et al., 2017). One of the

most popular and traditional TSC approaches is the use of a nearest neighbor

(NN) classiﬁer coupled with a distance function (Lines and Bagnall, 2015). Par-

ticularly, the Dynamic Time Warping (DTW) distance when used with a NN

classiﬁer has been shown to be a very strong baseline (Bagnall et al., 2017).

Lines and Bagnall (2015) compared several distance measures and showed

that there is no single distance measure that signiﬁcantly outperforms DTW.

They also showed that ensembling the individual NN classiﬁers (with diﬀerent

distance measures) outperforms all of the ensemble’s individual components.

Hence, recent contributions have focused on developing ensembling methods

that signiﬁcantly outperforms the NN coupled with DTW (NN-DTW) (Bag-

nall et al., 2016; Hills et al., 2014; Bostrom and Bagnall, 2015; Lines et al., 2016;

Schäfer, 2015; Kate, 2016; Deng et al., 2013; Baydogan et al., 2013). These

approaches use either an ensemble of decision trees (random forest) (Baydogan

et al., 2013; Deng et al., 2013) or an ensemble of diﬀerent types of discrimi-

nant classiﬁers (Support Vector Machine (SVM), NN with several distances) on

one or several feature spaces (Bagnall et al., 2016; Bostrom and Bagnall, 2015;

Schäfer, 2015; Kate, 2016). Most of these approaches signiﬁcantly outperforms

the NN-DTW (Bagnall et al., 2017) and share one common property, which is

the data transformation phase where time series are transformed into a new

feature space (for example using shapelets transform (Bostrom and Bagnall,

2015) or DTW features (Kate, 2016)). This notion motivated the development

of an ensemble of 35 classiﬁers named COTE (Collective Of Transformation-

based Ensembles) (Bagnall et al., 2016) that does not only ensemble diﬀerent

classiﬁers over the same transformation, but instead ensembles diﬀerent clas-

siﬁers over diﬀerent time series representations. COTE is currently considered

the state of the art for time series classiﬁcation (Bagnall et al., 2017) when

evaluated over the 85 datasets from the UCR archive (Chen et al., 2015b).

Deep learning for time series classiﬁcation: a review 3

To achieve its high accuracy, COTE becomes hugely computationally inten-

sive and impractical to run on a real big data mining problem (Bagnall et al.,

2017). The approach requires training 35 classiﬁers as well as cross-validating

each hyperparameter of these algorithms, which makes the approach infeasible

to train in some situations (Lucas et al., 2018). To emphasize on this infea-

sibility, note that one of these 35 classiﬁers is the Shapelet Transform (Hills

et al., 2014) whose time complexity is O(n

· l

) with n being the number

of time series in the dataset and l being the length of a time series. Adding

to the training time’s complexity is the huge classiﬁcation time of one of the

35 classiﬁers: the nearest neighbor which needs to scan the training set before

taking a decision at test time. Therefore since the nearest neighbor constitutes

an essential component of COTE, its deployment in a real-time setting is still

limited if not impractical. Finally, adding to the huge runtime of COTE, the

decision taken by 35 classiﬁers cannot be interpreted easily by domain experts,

since researchers already struggle with understanding the decisions taken by

an individual classiﬁer.

After having established the current state of the art of non deep classiﬁers

for TSC (Bagnall et al., 2017), we discuss the success of Deep Learning (Le-

Cun et al., 2015) in various classiﬁcation tasks which motivated the recent

utilization of deep learning models for TSC (Wang et al., 2017b). Deep Con-

volutional Neural Networks (CNNs) have revolutionized the ﬁeld of computer

vision (Krizhevsky et al., 2012). For example, in 2015, CNNs were used to reach

human level performance in image recognition tasks (Szegedy et al., 2015).

Following the success of deep neural networks (DNNs) in computer vision, a

huge amount of research proposed several DNN architectures to solve natu-

ral language processing (NLP) tasks such as machine translation (Sutskever

et al., 2014; Bahdanau et al., 2015), learning word embeddings (Mikolov et al.,

2013; Mikolov et al., 2013) and document classiﬁcation (Le and Mikolov, 2014;

Goldberg, 2016). DNNs also had a huge impact on the speech recognition com-

munity (Hinton et al., 2012; Sainath et al., 2013). Interestingly, we should note

that the intrinsic similarity between the NLP and speech recognition tasks is

due to the sequential aspect of the data which is also one of the main charac-

teristics of time series data.

In this context, this paper targets the following open questions: What is

the current state of the art DNN for TSC ? Is there a current DNN approach

that reaches state of the art performance for TSC and is less complex than

COTE ? What type of DNN architectures works best for the TSC task ? And

ﬁnally: Could the black-box eﬀect of DNNs be avoided to provide interpretabil-

ity? Given that the latter questions have not been addressed by the TSC

community, it is surprising how much recent papers have neglected the possi-

bility that TSC problems could be solved using a pure feature learning algo-

rithm (Neamtu et al., 2018; Bagnall et al., 2017; Lines et al., 2016). In fact, a

recent empirical study (Bagnall et al., 2017) evaluated 18 TSC algorithms on

85 time series datasets, none of which was a deep learning model. This shows

how much the community lacks of an overview of the current performance of

deep learning models for solving the TSC problem (Lines et al., 2018).

4 Hassan Ismail Fawaz et al.

123456789

t-LeNet

MCNN

MCDCNN

TWIESN

Time-CNN

MLP

Encoder

FCN

ResNet

Fig. 1: Critical diﬀerence diagram showing pairwise statistical diﬀerence com-

parison of classiﬁers for both univariate and multivariate time series classiﬁ-

cation archives.

In this paper, we performed an empirical comparative study of the most

recent deep learning approaches for TSC. With the rise of graphical processing

units (GPUs), we show how deep architectures can be trained eﬃciently to

learn hidden discriminative features from raw time series in an end-to-end

manner. Similarly to Bagnall et al. (2017), in order to have a fair comparison

between the tested approaches, we developed a common framework in Python,

Keras (Chollet, 2015) and Tensorﬂow (Abadi et al., 2015) to train the deep

learning models on a cluster of more than 60 GPUs.

In addition to the univariate datasets’ evaluation, we tested the approaches

on 12 Multivariate Time Series (MTS) datasets (Baydogan, 2015). The mul-

tivariate evaluation shows another beneﬁt of deep learning models, which is

the ability to handle the curse of dimensionality (Bellman, 2010; Keogh and

Mueen, 2017) by leveraging diﬀerent degrees of smoothness in compositional

function (Poggio et al., 2017) as well as the parallel computations of the

GPUs (Lu et al., 2015).

As for comparing the classiﬁers over multiple datasets, we followed the

recommendations in Demšar (2006) and used the Friedman test (Friedman,

1940) to reject the null hypothesis. Once we have established that a statistical

diﬀerence exists within the classiﬁers’ performance, we followed the pairwise

post-hoc analysis recommended by Benavoli et al. (2016) where the average

rank comparison is replaced by a Wilcoxon signed-rank test (Wilcoxon, 1945)

with Holm’s alpha correction (Holm, 1979; Garcia and Herrera, 2008). For

example, Figure 1 shows the critical diﬀerence diagram (Demšar, 2006), where

a thick horizontal line shows a group of classiﬁers (a clique) that are not

signiﬁcantly diﬀerent in terms of accuracy.

In this study, we have trained about 1 billion parameters across 97 univari-

ate and multivariate time series datasets. Despite the fact that a huge number

of parameters risks overﬁtting (Zhang et al., 2017) the relatively small train

set in the UCR archive (Chen et al., 2015b), our experiments showed that not

only DNNs are able to signiﬁcantly outperform the NN-DTW, but are also

able to achieve results that are not signiﬁcantly diﬀerent than COTE using a

deep residual network architecture (He et al., 2016; Wang et al., 2017b).

The rest of the paper is structured as follows. In Section 2, we provide some

background materials concerning the main types of architectures that have

been proposed for TSC. In Section 3, the tested architectures are individually

Deep learning for time series classiﬁcation: a review 5

presented in details. We describe our experimental open source framework

in Section 4. The corresponding results and the discussions are presented in

Section 5. In Section 6, we describe in details a method that mitigates the

black-box eﬀect of the deep learning models. Finally, we present a conclusion

in Section 7 to summarize our ﬁndings and discuss future directions.

The main contributions of this paper can be epitomized as follows:

– We explain with practical examples, how deep learning can be adapted to

one dimensional time series data.

– We propose a uniﬁed taxonomy that regroups the recent applications of

DNNs for TSC in various domains under two main categories: generative

and discriminative models.

– We detail the architecture of nine end-to-end deep learning models designed

speciﬁcally for TSC.

– We evaluate these models on the univariate UCR archive benchmark and

12 MTS classiﬁcation datasets.

– We provide the community with an open source deep learning framework

for TSC in which we have implemented all nine approaches.

– We investigate the use of Class Activation Map (CAM) in order to reduce

DNNs’ black-box eﬀect and explain the diﬀerent decisions taken by various

models.

2 Background

In this section, we start by introducing the necessary deﬁnitions for ease of un-

derstanding. We then follow by an extensive theoretical background on train-

ing DNNs for the TSC task. Finally we present our proposed taxonomy of the

diﬀerent DNNs with examples of their application in various real world data

mining problems.

2.1 Time series classiﬁcation

Before introducing the diﬀerent types of neural networks architectures, we go

through some formal deﬁnitions for TSC.

Deﬁnition 1 A univariate time series X = [x

, x

, . . . , x

] is an ordered set

of real values. The length of X is equal to the number of real values T .

Deﬁnition 2 An M -dimensional MTS, X = [X

, X

, . . . , X

] consists of T

observations X

∈ R

Deﬁnition 3 A dataset D = {(X

, Y

), (X

, Y

), . . . , (X

, Y

)} is a collec-

tion of pairs (X

, Y

) where X

could either be a univariate or multivariate

time series with Y

as its corresponding one-hot label vector.

The task of TSC consists of training a classiﬁer on a dataset D in order to

map from the space of possible inputs to a probability distribution over the

class variable values (labels).

剩余49页未读，继续阅读

评论收藏

内容反馈

hywcxq

粉丝: 0
资源: 33

Deep learning for time series classification a review.pdf

最新资源

Deep learning for time series classification a review.pdf

Deep Learning for the Classification

Deep Learning for Scene Classification A Survey.pdf

Learning Deep Features for One-Class Classification.pdf

Unsupervised Deep Transfer Feature Learning for Medical Image Classification

Deep Learning for Encrypted Traffic Classification:An Overview.pdf

classification papers for deep learning

Deep Learning for Scene Classification A Survey.zip

【2】A fast learning algorithm for deep belief nets.pdf

Deep_Learning_for_Aspect-Based_Sentiment_Analysis_A_Comparative_Review.pdf

deep-learning-for-signal-white-paper.pdf

【7】Deep residual learning for image recognition.pdf

Deep Learning for Chest X-ray Analysis A Survey.pdf

【船级社】 KR Rules for the Classification of Dredgers.pdf

Deep_Learning_for_Encrypted_Traffic_Classification_An_Overview.pdf

mini_course Deep_learning_time_series_forecasting_.pdf

deep learning经典论文30篇

Deep Learning for Beginners

timeseries cluster and classification

[machine_learning_mastery系列]deep_learning_with_python.pdf(with code)

Deep Time Series Forecasting with Python,pdf

Deep.Learning.with.Keras.epub

YOLOv8-deepsort 实现智能车辆目标检测+车辆跟踪+车辆计数

Transformer模型实现长期预测并可视化结果（附代码+数据集+原理介绍）

最新资源