《生成式对抗网络GAN时空数据应用》_时空数据处理与挖掘资源-CSDN文库

需积分: 50 26 浏览量 2020-08-23 01:27:49 上传评论 4 收藏 1.35MB PDF 举报

资源推荐

资源详情

资源评论

111

Generative Adversarial Networks for Spatio-temporal Data: A

Survey

NAN GAO, HAO XUE, WEI SHAO, SICHEN ZHAO, KYLE KAI QIN, ARIAN PRABOWO,

MOHAMMAD SAIEDUR RAHAMAN, and FLORA D. SALIM, RMIT University

Generative Adversarial Networks (GANs) have shown remarkable success in the computer vision area for

producing realistic-looking images. Recently, GAN-based techniques are shown to be promising for spatio-

temporal-based applications such as trajectory prediction, events generation and time-series data imputation.

While several reviews for GANs in computer vision been presented, nobody has considered addressing

the practical applications and challenges relevant to spatio-temporal data. In this paper, we conduct a

comprehensive review of the recent developments of GANs in spatio-temporal data. we summarise the

popular GAN architectures in spatio-temporal data and common practices for evaluating the performance

of spatio-temporal applications with GANs. In the end, we point out the future directions with the hope of

beneting researchers interested in this area.

CCS Concepts: •Computing methodologies → Machine learning; Articial intelligence;

Additional Key Words and Phrases: Generative adversarial nets, spatio-temporal data, time series, trajectory

data

ACM Reference format:

Nan Gao, Hao Xue, Wei Shao, Sichen Zhao, Kyle Kai Qin, Arian Prabowo, Mohammad Saiedur Rahaman,

and Flora D. Salim. 2020. Generative Adversarial Networks for Spatio-temporal Data: A Survey. ACM Comput.

Surv. 37, 4, Article 111 (August 2020), 28 pages.

DOI: 10.1145/1122445.1122456

1 INTRODUCTION

Spatio-temporal properties are commonly observed in various elds, such as transportation [

134

social science [

] and criminology [

125

], among which that have been rapidly transformed by

the proliferation of sensor and big data. e vast amount of spatio-temporal (ST) data requires

appropriate processing techniques for building eective applications. Generally, traditional methods

dealing with tabular data or graph data oen perform poorly when applied to spatio-temporal

datasets. e reasons are mainly three-folds [

145

]: (1) ST data are usually in continuous space

while tabular or graph data are oen discrete; (2) ST data usually present both spatial and temporal

properties where the data correlations are more complex to capture by traditional techniques; (3)

ST data tends to be highly self-correlated and data samples are usually not independently generated

as in traditional data.

With the prevalence of deep learning, many neural networks (e.g., Convolutional Neural Network

(CNN) [

], Recurrent Neural Network (RNN) [

], Autoencoder (AE) [

], Graph Convolutional

Network (GCN) [

]) have been proposed and achieved remarkable success for modelling ST

data. e wide adoption of deep learning for ST data is due to its demonstrated potential for

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee

provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and

the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored.

Abstracting with credit is permied. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires

prior specic permission and/or a fee. Request permissions from permissions@acm.org.

DOI: 10.1145/1122445.1122456

ACM Computing Surveys, Vol. 37, No. 4, Article 111. Publication date: August 2020.

arXiv:2008.08903v1 [cs.LG] 18 Aug 2020

111:2 Gao, et al.

hierarchical feature engineering ability. In this survey, we focus on one of the most interesting

breakthroughs in the deep learning eld - Generative Adversarial Networks (GANs) [

] and their

potential applications for ST data.

GAN is a generative model which learns to produce realistic data adversarially. It consists

of two components [

]: the generator

and discriminator

captures the data distribution

and produces realistic data from the latent variable

, and

estimates the probability of the data

coming from the real data space. GAN adopts the concept of the zero-sum non-cooperative game

where

and

are trained to play against each other until reaching a Nash equilibrium. GANs

have gained considerable aention in various elds, involving images (e.g., image translation [

super-resolution [

], joint image generation [

], object detection [

], change facial aributes

[

]), videos (e.g., video generation [

142

]), natural language processing (e.g., text generation [

text to image [159]).

However, applying image or video generation directly are not applicable for modelling ST data

such as trac ow, regional rainfall, and pedestrian trajectory. On one hand, image generation

usually takes the appearance between the input and output images into account, and fails to

adequately handle spatial variations. On the other hand, video generation considers spatial dynamics

between images, however, temporal changes are not adequately considered when the prediction of

the next image is highly dependent on the previous image [

130

]. Hence, new approaches need to

be explored for successfully applying GANs on ST data.

Recently, GANs have started being applied to ST data. e applications for GANs on ST data

mainly include the generation of de-identied spatio-temporal events [

130

], time series imputa-

tion [

], trajectory prediction [

], graph representation [

143

], etc. Despite the success

of GANs on computer vision area, applying GANs to ST data prediction is challenging [

130

]. For

instance, leveraging additional information such as Places of Interest (PoI), weather information

is still untouched in previous research. Besides, dierent to the images where researchers could

rely on visual inspections of the generated instances, evaluation of GANs on ST data remains an

unsolved problem. It is neither practical nor appropriate to adopt the traditional evaluation metrics

for GAN on ST data [33, 130].

A few research have reviewed recent literature on the problems in ST data or GAN applications

in dierent elds. Compared with mining paerns from traditional relational data, modelling

ST data is particularly challenging due to its spatial and temporal aributes in addition to the

actual measurements. Atluri et al. [

] have reviewed the popular problems and methods for

modelling ST data. A taxonomy of the dierent types of ST data, ways of dening and describing

data instances has been provided to identify the relevant problems for any type of ST data in

real-world applications. ey have also listed the commonly studied ST problems and reviewed

the issues for dealing with unique properties of dierent ST types. Want et al. [

145

] reviewed the

recent progress in applying deep learning to ST data mining tasks and proposed a pipeline of the

utilisation of deep learning models for ST data modelling problems. Hong et al. [

] explained the

GANs from various perspectives and enumerate popular GAN variants applied to multiple tasks.

Recent progress of GANs was discussed in [

111

] and Wang et al. [

146

] proposed a taxonomy of

GANs for computer vision area. Particularly, Yi et al. [

153

] reviewed recent advances of GANs in

medical imaging.

However, all the above works reviewed either ST data modelling problems or the recent progress

of GANs in the computer vision area. ough many researchers [

130

] have modelled

ST data with GANs, there is no related survey in this area to address the potential of using GANs

for ST data applications. For the rst time, this paper presents a comprehensive overview of GANs

ACM Computing Surveys, Vol. 37, No. 4, Article 111. Publication date: August 2020.

Generative Adversarial Networks for Spatio-temporal Data: A Survey 111:3

in ST data, describes promising applications of GANs, and identies some remaining challenges

needed to be solved for enabling successful applications in dierent ST related tasks.

To present a comprehensive overview of all the relevant research on GANs for ST data, we use

Google Scholar

to conduct automated keyword-based search [

123

]. According to [

], Google

Scholar provides coverage and accessibility, and digital libraries such as IEEE Explore

, Science

Direct

, ACM Digital Library

. e search period is limited from 2014 to 2020 (inclusive) as the

GAN has rst appeared in 2014 [

]. However, papers that introduce novel concepts or approaches

for spatio-temporal data mining can be predated 2014. To ensure that our survey covers all relevant

primary literature, we have included such seminal papers regardless of their publication date.

e remainder of the paper is organised as follows. In Section 2, we discuss the properties,

characteristics and common research problems of ST data. We also present the popular deep

learning methods with non-GAN frameworks for ST data, including the Convolutional Neural

Networks, Recurrent Neural Networks, Long Short-term Memory and Gated Recurrent Units. Section

3 reviews the denition of GANs and its popular variants with dierent architecture and loss

functions. Section 4 lists the recent research progress for GANs in dierent categories of ST

applications. Section 5 summarises the challenges on processing ST data with GANs, including the

adapted architectures, loss functions and evaluation metrics. Finally, we conclude the paper and

discuss future research directions.

2 PRELIMINARY

2.1 Spatio-temporal Data

e existence of time and space introduces a rich variety of spatio-temporal data types, leading

to dierent ways of formulating spatio-temporal data mining problems and techniques. In this

part, we will rst introduce the general properties of spatio-temporal data, then briey describe

the common types of spatio-temporal data in dierent applications using generative adversarial

nets techniques.

2.1.1 Properties. ere are several general properties for spatio-temporal data (i.e., spatial

reference, time reference, auto-correlation and heterogeneity [10]) described as below.

Spatial Reference

. e spatial reference describes whether the objects are associated with the

xed location or dynamic locations [

]. Traditionally, when the data is collected from stationary

sensors (e.g., weather stations), we consider the spatial dimension of the data is xed. Recently,

with the boost of mobile computing and location-based services, the dynamic locations of moving

objects have been recorded where the collected data comes from sensors aached to dierent

objects, e.g., GPS trajectories from road vehicles [115].

Temporal Reference

. e temporal reference describes to what extent the objects evolve [

e simplest context includes objects do not evolve at all where only the static snapshots of objects

available. In a slightly more complicated situation, objects can change status but only the most

recent update snapshot remains where the full history of status is unknown. e extreme context

consists of moving objects where the full history of moving is kept, therefore generating time series

where all the status have been traversed.

Auto-correlation

. e observations of spatio-temporal data are not independent and usually

have spatial and temporal correlations between near measurements. For example, in the trans-

portation area, sensors in each parking lot with the unique spatial location can record the temporal

hps://scholar.google.com/

hps://ieeexplore.ieee.org/

hps://www.sciencedirect.com/

hps://dl.acm.org/

ACM Computing Surveys, Vol. 37, No. 4, Article 111. Publication date: August 2020.

111:4 Gao, et al.

(a) Spatio-temporal events

(b) Two trajectories

Fig. 1. Examples of spatio-temporal events and trajectories

information when a vehicle arrives or leaves [

116

134

]. is auto-correlation of spatio-temporal

data results in the smoothness of temporal measurements (e.g., temperature changes smoothly

over time ) and consistency between the spatial measurements (e.g., temperature values are similar

in adjacent locations). ereby, the traditional GAN techniques for computer vision eld (e.g.,

image generation [

]) without considering the temporal correlation may not well suited for the

spatio-temporal data.

Heterogeneity

. Spatio-temporal dataset can show heterogeneity in spatial or temporal infor-

mation on dierent levels. For instance, trac ow in a city can show similar paerns between

dierent weeks. During a week, the trac data on Monday may be dierent from data on Friday.

ere can also be inter-week changes due to public events or extreme weather, which can aect

the trac paerns in a city. To deal with the heterogeneity of spatial and temporal information, it

is necessary to learn dierent models for dierent spatio-temporal regions.

2.1.2 Data Types. ere are various spatio-temporal data types in real-world applications,

diering in the representation of space and time context [

]. We describe the four common types

of spatio-temporal data which have been studied with GAN recently: (1) time series [

101

166

]; (2) spatio-temporal events [

116

130

134

]; (3) spatio-temporal graphs

[

143

152

]; (4) trajectory data [

]. In this part, we provide a taxonomy of the data types available

in the spatio-temporal domain, then briey discuss the properties of those data types and potential

diculties when facing with GANs.

Time Series

. A time series can be represented as a sequence of data points

X = {X

, X

, . . . , X

}

listed in an order of time (i.e., sequence of discrete-time data [

140

]). Examples of time series include

the values of indoor temperature during a day [

119

], the changes of accelerometer readings in

the IoT devices [

], uctuations of the stock price in a month [

166

], etc. Time series analysis

consists of techniques to analyse time series for extracting useful statistic information and other

characteristics of data. e common questions that used for dealing with time series include but not

limited to: Can we predict the future values for time series based on the historical values [

105

147

Can we cluster groups of time series with similar temporal and spatial paerns [

]? Can we impute

the missing values automatically in multi-variate time series [

102

]? Can we split time series into

dierent segments with its own characteristic properties [28, 63]?

Spatio-temporal Events

. An spatio-temporal event represents a tuple containing temporal,

spatial information as well as an additional observed value [

]. Generally, it is denoted as

, t

, l

}

, where

and

indicates the time and location of the event,

means the value to

describe the event. Typically, the locations are recorded in three dimensions (i.e., latitude, longitude,

ACM Computing Surveys, Vol. 37, No. 4, Article 111. Publication date: August 2020.

Generative Adversarial Networks for Spatio-temporal Data: A Survey 111:5

Fig. 2. Example of Spatio-temporal Graph Data

and altitude or depth), although sometimes only 1 or 2 spatial coordinates are available. Spatio-

temporal events (see Fig. 1(a)) are frequently used in real-world applications such as the taxi demand

[

118

], trac ow [

130

], urban crimes [

124

], forest res[

], etc. In some cases, spatio-temporal

events may even have duration like parking or heliophysics [

113

]. Usually, an ordered set of

spatio-temporal events can also be considered as an trajectory where the spatial locations visited by

moving objects. Some common questions that used for analysing spatio-temporal events includes:

Can we predict the future spatio-temporal events based on the previous observations [

130

]? How

are spatio-temporal events clustered based on time and space [

135

]? Can we identify the anomalous

spatio-temporal events that do not follow the common paers of other events [12]?

Trajectory data

. A trajectory represents the recordings of locations of a moving object at

certain times and it is usually dened as a function mapped from the temporal domain to the

spatial domain [

]. Trajectories of moving points can be denoted as a sequence of tuples

P = {(x

, y

, t

), (x

, y

, t

), . . ., (x

, y

, t

)}

, where

, y

, t

)

indicates the location

, y

)

at time

Several research have been conducted in the eld of trajectory data mining and there are four major

categories [

163

]: mobility of people [

120

], mobility of transportation [

130

], mobility of natural

phenomena and mobility of animals [

]. Fig. 1(b) shows an example of two trajectories of object

and object

. e common questions for processing trajectory data include: Can we predict the

future trajectory based on the historical trajectory traces [

127

128

]? Can we divide a collection

of trajectories into small representative groups [

133

]? Can we detect the abnormal behaviours from

trajectories [89]?

Spatio-temporal Graph

. Spatio-temporal graph structure provides the representation of the

relations between dierent nodes in dierent time. A sequence of spatio-temporal graphs [

152

]

can be represented as

G = (G

, G

, . . . , G

)

where

= {V

, E

, W

}

indicates the graph snapshot

at time

(

i ∈ {

, . . . , n}

). Spatio-temporal graphs have been applied in various domains such

as commerce (e.g., trades between countries [

]), transportation (e.g., route planning algorithms

[

], trac forecasting [

155

]) and social science (e.g., studying geo-spatial relations of dierent

social phenomena [

]). Fig. 2 is an example of spatio-temporal graphs in

, T

. Some common

questions for processing spatio-temporal graph includes: Can we forecast the status of graph based

on the historical graph representations [

143

155

] ? Can we predict the links based on the previous

graph networks [77]?

2.2 Spatio-Temporal Deep Learning with Non-GAN Networks

is section introduces the traditional deep learning approaches for spatio-temporal data mining

with Non-GAN networks, including Convolutional Neural Network, Recurrent Neural Network,

Autoencoder, Graph Convolutional Network etc.

ACM Computing Surveys, Vol. 37, No. 4, Article 111. Publication date: August 2020.

剩余27页未读，继续阅读

评论收藏

内容反馈

syp_net

粉丝: 158
资源: 1196

《生成式对抗网络GAN时空数据应用》

生成对抗网络（GAN）实例 代码+数据集

生成对抗网络(GAN)数据回归预测，GAN回归预测，多输入单输出模型(Matlab完整源码和数据）

生成对抗网络GAN.pptx

基于DEAP数据集的脑电情绪识别（构建生成对抗网络（GAN）和条件GAN（CGAN）模型）Pytorch框架

GAN-生成式对抗网络

TimeGAN:时间序列生成对抗网络（TimeGAN）的代码库-NeurIPS 2019

（gan神经网络生成一维数据）gan.py

LSTM_GAN修改.ipynb

基于GPS轨迹和照片轨迹 进行时空数据挖掘

Real-Time-Facial-Expression-Recognition-with-DeepLearning

生成式对抗网络（GAN）系列

针对MNIST数据集训练的生成对抗网络（GAN）.zip

生成式对抗网络GAN.ipynb

生成式对抗网络GAN 综述

生成式对抗网络GAN综述

生成对抗网络（GAN）代码+数据集

GANLab生成对抗网络GAN的交互式可视化实验工具

tensorflow实现的gan

时空序列预测建模

生成式对抗网络在图像补全中的应用

基于生成式对抗网络(GAN)的自动驾驶容错感知研究

41.生成式对抗网络（GAN）基础 - 知乎1

GAN对抗式生成网络的应用：从图片上生成诗歌

生成式对抗网络GAN的研究进展与展望_王坤峰.pdf

李宏毅GAN课程讲义、生成式对抗网络课件

最新《生成式对抗网络GAN逆转》综述论文

GAN（生成式对抗网络）读书笔记（上）

最新《生成式对抗网络GAN进展》综述论文

生成式对抗网络GAN的研究进展与展望

最新资源

生成对抗网络（GAN）实例代码+数据集

基于GPS轨迹和照片轨迹进行时空数据挖掘