mm姿势：使用mmWave雷达和CNN的实时人体骨骼姿势估计1

1558-1748 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JSEN.2020.2991741, IEEE Sensors

Journal

IEEE SENSORS JOURNAL, VOL. XX, NO. XX, XXXX 2020 1

mm-Pose: Real-Time Human Skeletal Posture

Estimation using mmWave Radars and CNNs

Arindam Sengupta, Student Member, IEEE, Feng Jin, Student Member, IEEE,

Renyuan Zhang, Student Member, IEEE, and Siyang Cao, Member, IEEE

Abstract—In this paper, mm-Pose, a novel approach to detect

human skeleton for effective and preventive decision making

in real-time. The use of radar makes the system operationally

robust to scene lighting and adverse weather conditions. The

reflected radar point cloud in range, azimuth and elevation

are first resolved and projected in Range-Azimuth and Range-

Elevation planes. A novel low-size high-resolution radar-to-image

representation is also presented, that overcomes the sparsity

in traditional point cloud data and offers significant reduction

in the subsequent machine learning architecture. The RGB

channels were assigned with the normalized values of range,

Swinging both arms to validate accurate predictions for motion

in range, azimuth and elevation. The detailed methodology,

implementation, challenges, and validation results are presented.

Index Terms—Convolutional Neural Networks, mmWave

Radars, Posture Estimation, Skeletal Tracking

W

(CV) has emerged as an exciting field of research to pro-

object classification, target tracking, traffic monitoring and

autonomous vehicles [3]–[7]. In the recent years, another

interesting topic that the CV community has been exploring

is the ability to estimate human skeletal pose by identifying

and detecting specific joints and/or body parts from still/video

data. This specific area of research finds several applications,

one being primarily in the health-care industry by automating

patient monitoring systems, with the current situation of global

shortage in nursing staff [8]. Such tracking systems would

also allow for effective pedestrian monitoring for autonomous

external sources for illuminating the scene and are there-

fore rendered ineffective in poor lighting conditions, adverse

weather conditions or when the scene/target is occluded [9].

These could result in irrevocable catastrophic events similar

to the ones encountered at (i) Tesla’s autopilot testing, where

the vision sensors failed to detect the white side of a tractor

trailer in brightly lit sky (very high reflectivity) [10] , and (ii)

Uber self-driving vehicle crash incident in Arizona due to the

vision/LiDAR sensors’ inability to detect the pedestrian in time

to avoid the accident during a night test (low/no reflectivity)

1558-1748 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JSEN.2020.2991741, IEEE Sensors

Journal

2 IEEE SENSORS JOURNAL, VOL. XX, NO. XX, XXXX 2020

resent the scene using RF intensity based spatial heat-maps,

reflection point clouds or range-doppler maps, rather than a

true-color image representation. Radars are therefore primarily

used for target localization and tracking applications. Further-

more, object classification becomes non-trivial with radar data,

and the lack of available labeled radar data-sets for this task

makes it even more challenging.

Traditionally, radar systems have been size and cost inten-

sive primarily targeted to commercial and defense applica-

compact and are extremely practical to deploy. Furthermore,

mmWave radars provide us with a high resolution point cloud

representation of the scene and have therefore emerged as one

of the primary sensors in autonomous robots on a smaller

scale, to more commercial applications such as autonomous

vehicles. Higher operating bandwidths also allow mmWave

radars to roughly generate the contour of human body without

extracting facial information, thus preserving user privacy.

In this paper, we propose mm-Pose, a novel real-time

approach to estimate and track human skeleton using mmWave

find applications in (i) privacy-protected automated patient

monitoring systems, and (ii) aiding defense forces in a hostage

situation. Radars carrying this technology on unmanned aerial

vehicles (UAVs) could scan the building and map the live

skeletal postures of the hostage and the adversary, through

the walls, which would not have been possible otherwise with

vision sensors.

The paper is organized as follows. Section II summarizes

the current skeleton tracking work carried out in the CV com-

munity and its extension to RF sensors. A concise background

study is summarized and concluded in Section VI.

II. LITERATURE REVIEW

It is extremely critical to accurately estimate and track

human posture in several applications, as the estimated pose

is key to infer their specific behavior. Since the last decade,

using rectangular templates from RGB images/videos [15]. A

2016, that uses predicted torso keypoint activations to detect

multiple persons using agglomerative clustering [16]. Another

the skeleton from images, using a ResNet variant architecture

[17]. In 2016, DeeperCut, an improved multi-person pose

estimation model from DeepCut was proposed that used a

bottom up approach using a fine-tuned ResNet architecture

by researchers at Carneigie Mellon University in 2017 [21].

as previously introduced, privacy concerns restrict the use of

1558-1748 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JSEN.2020.2991741, IEEE Sensors

Journal

SENGUPTA, A. et al.: MM-POSE: REAL-TIME HUMAN SKELETAL POSTURE ESTIMATION USING MMWAVE RADARS AND CNNS (APRIL 2020) 3

[25], [26] as the signatures solely represented the temporal

velocity profiles of the reflection points. Skeleton tracking

using RF signals is a new and emerging area of research.

RF based devices can be further classified into two categories,

wearable and non-wearable. Wearable wireless sensors use Wi-

Fi signals to track the location and velocity of the device,

which indirectly represents the human. However, Wi-Fi sig-

nals cannot distinguish between different body parts and are

therefore not suited for the proposed task of pose estimation.

Non-wearable RF sensors, such a radar, can be traditionally

A teacher-student encoder-decoder architecture was used to

of the chirps during one CPI.

rate. The echo from a target is a time delayed version of

the transmitting chirp. After stretch processing, the resulting

baseband signal is given as:

velocity. The radar cross section (RCS), which represent the

reflectivity of the target, can be solved by:

The equation above shows the relationship between the

The beat frequency resolution, that depends on the sampling

time in one chirp, is expressed as:

after ignoring the smaller multiplicative term, and

is expressed as:

depends on the sampling interval in one CPI, and is repre-

sented as:

Using (9) and (10), the obtained velocity resolution is

expressed as: