JointActivityRecognitionandIndoor_wifi

共1个文件

pdf：1个

版权申诉

wifi

124 浏览量 2021-10-04 02:04:57 上传评论收藏 1.63MB ZIP 举报

资源详情

资源评论

资源推荐

收起资源包目录

Joint Activity Recognition and Indoor.zip （1个子文件）

Joint Activity Recognition and Indoor.pdf 1.87MB

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.

Digital Object Identiﬁer 10.1109/ACCESS.2019.DOI

Joint Activity Recognition and Indoor

Localization with WiFi Fingerprints

FEI WANG

1,2

, JIANWEI FENG

, YINLIANG ZHAO

, XIAOBIN ZHANG

, SHIYUAN ZHANG

AND JINSONG HAN

3,1

Department of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, Shaanxi 710049 China

The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213 USA

Institute of Cyberspace Research, Zhejiang University, Hangzhou, Zhejiang 310058 China

Corresponding author: Yinliang Zhao (e-mail: yinliangzhao1960@gmail.com).

ABSTRACT Recent years have witnessed the rapid development in the research topic of WiFi sensing

that automatically senses human with commercial WiFi devices. This work falls into two major categories,

i.e., the activity recognition and the indoor localization. The former work utilizes WiFi devices to recognize

human daily activities such as smoking, walking, and dancing. The latter one, indoor localization, can

be used for indoor navigation, location-based services, and through-wall surveillance. The key rationale

behind this type of work is that people behaviors can inﬂuence the WiFi signal propagation and introduce

speciﬁc patterns into WiFi signals, called WiFi ﬁngerprints, which can be further explored to identify

human activities and locations. In this paper, we propose a novel deep learning framework for joint activity

recognition and indoor localization task using WiFi Channel State Information (CSI) ﬁngerprints. More pre-

cisely, we develop a system running standard IEEE 802.11n WiFi protocol, and collect more than 1400 CSI

ﬁngerprints on 6 activities at 16 indoor locations. Then we propose a dual-task convolutional neural network

with 1-dimensional convolutional layers for the joint task of activity recognition and indoor localization.

Experimental results and ablation study show that our approach achieves good performances in this joint

WiFi sensing task. Data and code have been made publicly available at https://github.com/geekfeiw/apl.

INDEX TERMS CSI Fingerprints, Activity Recognition, Indoor Localization, Human-Computer Interac-

tion, 1D Convolutional Neural Networks

I. INTRODUCTION

Channel State Information of WiFi devices have been ex-

tensively explored for human sensing tasks such as activity

recognition [1]–[4], gesture recognition [5]–[8], indoor lo-

calization [9]–[12], and health-care applications [13]–[19].

This prosperity beneﬁts from several special properties of

WiFi, including the ubiquitous deployment of commercial

WiFi devices, the robustness to lighting condition and occlu-

sion which overcomes limitation of cameras, and the non-

intrusiveness sensing which requires no user’s extra effort.

Though there is abundant work on the speciﬁc aforemen-

tioned WiFi human sensing task [3], [4], [6], [7], [13]–

[18], [20], little work aims at completing the joint task of

activity recognition and indoor localization. Carrying out the

joint task would breed numerous useful human-computer

interaction applications. For example, in a smart home with

Internet-of-Things (IoT) devices [21], [22], the devices could

precisely response differently to the same gesture command

based on user’s location. More speciﬁcally, the user can use

the gesture of ‘hand down’ to turn down the television in front

of her, whereas she can also use the same gesture to lower

the air conditioner’s temperature when standing close to the

air conditioner. To our best knowledge, MultiTrack [23] is

the only work that enables indoor localization and activity

recognition jointly, however it requires high-end hardware

modiﬁcation for ultra-wide band WiFi (over 600MHz).

The joint task can be summarized as the following two

folds. (1) Recognizing activities conducted at different lo-

cations. (2) Localizing the user by the activities. However,

there are two major challenges lying in the way. The ﬁrst

challenge is that WiFi ﬁngerprint differs even when perform-

ing a same activity but at different locations, thus we need to

look for a same representation for activities conducted at all

locations. The second one is that WiFi ﬁngerprints vary when

performing different activities in one location, thus we have

to explore distinguished features for each location from the

ﬁngerprint variances.

To conclude the above challenges formally, WiFi ﬁnger-

VOLUME 4, 2019 1

arXiv:1904.04964v2 [cs.HC] 18 Jul 2019

Fei Wang et al.: Joint Activity Recognition and Indoor Localization with WiFi Fingerprints

print, W , contains two components at the same time, activity

category (A) and user location (L). We denote the WiFi

ﬁngerprint as W (A, L). Joint activity recognition and indoor

localization task aims to learn a function f , which is capable

to classify activity categories (f : W (A, L) → A) and to

localize the user (f : W (A, L) → L), simultaneously. Thus

we formalize the joint task as f : W (A, L) → (A, L).

To this end, in the paper we propose a novel 1-

dimensional Convolutional Neural Network (C1D) including

two branches, one for activity recognition and the other

for indoor localization. To date, conventional 2-dimensional

Convolutional Neural Networks (C2D), which have brilliant

ability to learn features from raw data, boost the development

of computer vision [24]–[29], robotics [30]–[33], machin-

ery [34]–[36], etc. Unlike C2D that processes 2D spatial

data such as images, C1D is capable to process 1D temporal

data. For temporal WiFi ﬁngerprints, we design a C1D based

on the ResNet [24] to carry out the joint task of activity

recognition and indoor localization.

To evaluate our proposed approach, we implement the

standard IEEE 802.11n protocol in two universal software

radio peripheral (USRP) sets, Ettus N210

, where one Ettus

N210 broadcasts WiFi signals and the other parses Channel

State Information (CSI) ﬁngerprints of WiFi for joint task.

We deﬁne 6 hand gestures for potential human-computer

interaction applications, namely, hand up, hand down, hand

left, hand right, hand circle and hand cross. One volunteer

repeats these activities 15 times at each location (16 locations

in all) and forms a dataset with 1394 samples (after excluding

the invalid data). We evaluate our proposed C1D on this

dataset and present the results with several metrics such as

confusion matrix, F1 scores, convolutional feature maps, etc.

Experimental results show our proposed C1D achieves a very

promising performance in the joint task. We summarized our

contributions as follows.

1. We ﬁrst propose and achieve the joint task of activity

recognition and indoor localization, which enable practical

user gesture control in smart homes for human-computer

interaction applications.

2. We novelly view CSI ﬁngerprints as time series with

channel dimension and time dimension, apply a advanced

1-dimensional Convolutional Neural Network that sweeps

along the time dimension of the CSI ﬁngerprints, and achieve

the joint task of activity recognition and indoor localization.

3. We implement IEEE 802.11n protocol in two USRP sets

and build a dataset speciﬁcally for the joint task. We evaluate

the performance of proposed deep network on this dataset

and fully analyze the results.

II. RELATED WORK

A. CSI FINGERPRINTS

CSI ﬁngerprints of WiFi have been widely utilized for activ-

ity recognition [3], [4], [6], [7], [13], [15] and indoor localiza-

tion [37]–[40]. As for activity recognition, in [3], [13], [15],

https://www.ettus.com/all-products/un210-kit/

CSI ﬁngerprints are used to detect user falling especially for

the elderly-care system. In [7], CSI ﬁngerprints are used to

infer user keystroke. Further in [41], researchers ﬁnd that

CSI ﬁngerprints can reveal people’s typing when they use

smart phones in public WiFi. In [6], [42] CSI ﬁngerprints

are designed for hand sign recognition for human-computer

interactions. As for indoor localization, [37]–[40] collect

CSI ﬁngerprints corresponding to people locations, and train

classiﬁers to localize people with collected CSI ﬁngerprints.

To our best knowledge, there is no work on joint activity

recognition and indoor localization, which is very useful in

controlling different smart devices at different locations with

a set of pre-deﬁned activities. We achieve this task by a dual-

branch Convolutional Neural Network.

B. CSI FINGERPRINTS CLASSIFICATION

There exist three popular approaches in CSI ﬁngerprints

classiﬁcation. (1) Hand-crafted features + Support Vector

Machine (SVM) [43]: [13], [15] apply statistical values of

CSI time-series such as the mean, maximum, minimum,

entropy, etc., as features to train SVM with kernel methods

for CSI ﬁngerprints classiﬁcation. This approach requires

expertise in designing features, which is even much harder

on joint activity recognition and indoor localization. (2) Dy-

namic Time Wrapping (DTW) + k Nearest Neighbors (kNN):

[3], [6], [42] ﬁrst build a dataset with CSI ﬁngerprints.

When classifying a test CSI sample, this approach requires

computing all distances between the test sample and all

samples in the dataset, which is time-consuming compared

to pre-training a classiﬁer ﬁrst. (3) Deep learning: [37],

[40] utilizes deep Boltzmann Machine (DBM) to do indoor

localization. However DBM relies heavily on careful design

and tricks to converge. [38], [39] apply 3-5 convolutional

layers on activity recognition. In general, the shortage in the

depth limits the performance. [42] utilizes ResNet [24] and

Inception [26] to categorize CSI ﬁngerprints, whereas it only

handles single moment CSI, i.e., rather than handling tem-

poral CSI ﬁngerprints. In this paper, we propose a ResNet-

based Convolutional Neural Network to do CSI ﬁngerprints

classiﬁcation.

C. 1D CONVOLUTIONAL NEURAL NETWORK

Conventional Convolutional Neural Network (C2D) [24]–

[26] are designed for 2D inputs such as images. C2D applies

2D convolutional kernels to sweep along the width and height

of an image to capture its semantic and structural informa-

tion for image classiﬁcation [24]–[26], object detection [44],

instance segmentation [27], etc. In [45], [46], researchers

apply 3D CNN (C3D) on video data, which sweeps along the

width, height, and time of the video to capture information

both in spatial and in temporal. In this paper, we apply 1D

convolutional kernels to sweep along the time axis of the CSI

ﬁngerprint series to capture the temporal information of CSI

ﬁngerprints, which works well in the joint task of activity

recognition and indoor localization.

2 VOLUME 4, 2019

Fei Wang et al.: Joint Activity Recognition and Indoor Localization with WiFi Fingerprints

mother board

daughter board

antenna

sync

cable

ethernet cable

Ettus Clock

FIGURE 1: Main hardwares: Ettus USRP N210 and Ettus Clock.

PC-Tx PC-RxUSRP-Tx USRP-Rx

Ettus Clock

antenna-Tx

antenna-Rx

ethernet cable

sync cable

FIGURE 2: System framework. The system contains two sets

of personal computers and USRPs, which work as the WiFi

transmitter and receiver, respectively. An Ettus clock syn-

chronizes the two sets.

III. DATA COLLECTION

A. HARDWARE

We implement the standard IEEE 802.11n protocol in two

universal software radio peripherals (USRPs) to collect CSI

ﬁngerprints. As shown in FIGURE 1, the ﬁrst two ﬁgures

are the top view and front view of the USRP (Etuss N201),

respectively. The USRP is mainly composed of a mother

board, a daughter board and a WiFi antenna, which is used

to broadcast or receive WiFi signals under the control of

GNU Radio

. The details are listed below. Meanwhile, the

assembling diagram is shown in FIGURE 2.

1. Etuss N210s: A hardware with ﬁeld programmable gate

array (FPGA) that can be embedded IEEE 802.11n protocol

to send and receive WiFi packages for CSI ﬁngerprints.

2. Etuss Clock

and synchronization cables: Synchroniz-

ing N210s with GPS clock to avoid a WiFi phase shifting

caused by the clock differences between two N210s.

3. Antennas: To broadcast or receive WiFi signals under

the control of GNU Radio

4. Computers and Ethernet cables: To control N210s when

are set in a same local area network as N210s.

B. ACTIVITY AND LOCATION

We design 6 activities, namely, hand up, hand down, hand

left, hand right, hand circle, and hand cross, for human-

computer interaction applications, as shown in FIGURE 3.

This cluster of activities covers the majority of daily com-

mands for smart Internet-of-Things home, where using cam-

https://www.gnuradio.org/

https://www.ettus.com/all-products/OctoClock-G/

https://www.gnuradio.org/

FIGURE 3: Six gesture commands mainly for human-computer

interaction applications in smart home, i.e., hand circle, hand

up, hand cross, hand left, hand down, and hand right.

door

#2#1 #4#3

#6#5 #8#7

#10#9 #12#11

#14#13 #16#15

USRP-RxUSRP-Tx

0.8𝑚

FIGURE 4: One volunteer does activities at 16 locations.

eras are not practical due to security and privacy concerns.

Here we illustrate how our proposed activities work by the

case of television. “Hand up” and “hand down” can be used

to turn up and down the voice volume, respectively; “Hand

left” and “hand right” indicate switching channels; “Hand

circle” and “hand cross” are for CONFIRM command and

CANCEL command, respectively.

Besides recognizing activity in smart home, localizing

the user when s/he is doing an activity is also crucial for

the joint task. By combining user’s activity and location

together, we are able to infer user’s intention more precisely

and make it possible for users to control a range of smart

VOLUME 4, 2019 3

Fei Wang et al.: Joint Activity Recognition and Indoor Localization with WiFi Fingerprints

Stand still Play action

Sampled CSI Series Sampled CSI Series Sampled CSI Series

CSI Amplitude

FIGURE 5: Signal preprocessing. we manually annotate the action duration (leftmost), split CSI samples during action (middle),

and upsample the splitted CSI series to be size of 192 (rightmost). (take the 29th subcarrier series when the volunteer plays a

‘circle’ action at #2 position for example.)

Sampled CSI Series Sampled CSI Series

Sampled CSI Series

CSI Amplitude

Sampled CSI Series Sampled CSI Series

Sampled CSI Series

CSI Amplitude

circle at #10 up at #10 cross at #10

left at #10 down at #10

right at #10

FIGURE 6: CSI ﬁngerprint samples of 6 activities on #10

position.

Sampled CSI Series Sampled CSI Series

Sampled CSI Series

CSI Amplitude

circle 1 at #3 circle 2 at #3 circle 3 at #3

FIGURE 7: Do ‘circle’ at the #3 position. CSI samples vary in

proﬁles. CSI may be partly captured because of late action

start.

Sampled CSI Series Sampled CSI Series

Sampled CSI Series

CSI Amplitude

circle at #6 circle at #9

circle at #15

FIGURE 8: Do ‘circle’ at the #6 (left), the #9 (middle), and the

#15 position.

devices with the same activity. For example, the user may

want to communicate with the television when sitting on

the sofa, whereas s/he probably needs to control the air

conditioner (AC) when standing in front of an AC. To make

a proof-of-concept experiment, we collect CSI ﬁngerprints

when a volunteer does 6 activities (shown in FIGURE. 3) at

16 locations in one room. We illustrate the spatial relationship

in FIGURE. 4, where the 16 locations are selected evenly

with a purpose to cover most central area of the room. The

USRPs are ﬁxedly placed besides the selected locations. In

all, the volunteer repeats each of the 6 activities 15 times at

16 locations and forms a dataset with 1440 samples.

C. CSI FINGERPRINT ANALYSIS

We visualize some samples varying in activities and locations

to present the challenges of joint task of activity recognition

and indoor localization. FIGURE 6 shows CSI ﬁngerprints

when the volunteer does 6 activities at #10 location, in

which the x-axis is the sampling index (time) and the y-

axis is the amplitude of the CSI ﬁngerprints. There are 52

time series in each CSI ﬁngerprints, differed with 52 colors

in FIGURE 6. 52 is the number of orthogonal frequency

division multiplexing (OFDM) [47] sub-carriers that carry

data in parallel in WiFi protocol. FIGURE 6 demonstrates

that CSI ﬁngerprints vary when the user conducts 6 activities

at a same location.

FIGURE 7 illustrates 3 CSI ﬁngerprint samples when the

volunteer carries out “hand circle” at #3 location. FIGURE 7

shows that though the volunteer plays the same activity at the

same location, CSI ﬁngerprints are still very different in time-

serial proﬁle (left and middle), and in the start point of the

activity (left and right). Besides, performing the same activity

at different locations also largely varies CSI ﬁngerprints as

illustrated in FIGURE 8, making it challenging to ﬁnd out

shared features for one activity at all locations.

IV. METHODOLOGY

A. PREPROCESSING

As shown in FIGURE 2, we have one transmitting antenna

and one receiving antenna. As shown in FIGURE 7, the num-

ber of sub-carriers and the streams for each CSI ﬁngerprint

are 52 and 192, respectively. This setting makes each CSI

ﬁngerprint a matrix with size of 52 × 192. Note that we only

use CSI amplitude and ignore the CSI phase for the fusion of

both is out of range of this paper. Because CSI ﬁngerprints

vary according to different activity start time and ﬁnish time,

we manually annotate the activity duration to prepare useful

signals for further use. We take the time series of 29th sub-

carrier as the visualization example to show a duration an-

notation in FIGURE 5 (left). This annotating process enables

us to directly use the segmented CSI ﬁngerprints for the joint

task. We then upsample the segmented CSI ﬁngerprints to

make them the same size using the linear interpolation (in

our experiment, the sizes of original and upsampled CSI

ﬁngerprints are both 192). One interpolated sample is shown

in FIGURE 5 (right).

4 VOLUME 4, 2019

评论收藏

内容反馈

版权申诉

Dyingalive

粉丝: 88
资源: 4808

Joint Activity Recognition and Indoor_wifi_源码

评论0

最新资源

Joint Activity Recognition and Indoor_wifi_源码

评论0

matlab代码步骤-WIFI-fingerprint-indoor-localization:WIFI-指纹-室内-定位

Device-Free Indoor Activity Recognition System

dlib_face_recognition_resnet_model_v1.dat

dlib_face_recognition_resnet_model_v1.zip

dlib_face_recognition_resnet_model_v1.dat.zip

dlib_face_recognition_resnet_model_v1.dat.zip_ResNet_dlib_face r

dlib_face_recognition_resnet_model_v1.dat人脸识别模型

Pattern_Recognition_and_machine_learningBishop_2006_PRML

dlib_face_recognition_resnet_model_v1和shape_predictor_68_face_landmarks

Pattern_Recognition_and_Machine_Learning

Pattern_Recognition_and_Big_Data

GART.rar_Activity Recognition_GART_gesture_手势库_隐马尔可夫

voicerecognition_MFCCmatlab_MFCC_说话人识别_tentd1i_MATLABMFCC_源码.zip

A_survey_of_named_entity_recognition_and_classification.pdf

joint_bayesian.rar_Joint Bayesian_face recognition_人脸识别_人脸识别、_联合

人脸识别程序C#版源码.rar_face recognition_人脸_人脸合成 c#_人脸识别

FACE RECOGNITION SYSTEM 2.1__faceprotected2

HSI_recognition-master_nicembc_hsi_高光谱_高光谱图像分类_高光谱图像_源码.zip

Emotion-recognition-master_脑电_python脑电_脑电信号_bowcw8_心电情感_源码.zip

冰河的渗透实战笔记-冰河.pdf

大灰狼远控2021最新版，解压密码222

J-LINK V10 V11固件.rar

ISO21434.pdf

Web安全漏洞扫描工具-AWVS14

CTF 竞赛入门指南（ctf-all-in-one）.pdf

stm32f103 adc采样+dma传输+fft处理 频率计_fft处理_stm32_ADCFFT_频率计_ADC采样_

Web中间件常见漏洞总结.pdf

jts-1.14.zip

CobaltStrike4.4.zip

最新资源

stm32f103 adc采样+dma传输+fft处理频率计_fft处理_stm32_ADCFFT_频率计_ADC采样_