Occlusion Aware Facial Expression Recognition Using CNN With Att...

IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 28, NO. 5, MAY 2019 2439

Occlusion Aware Facial Expression Recognition

Using CNN With Attention Mechanism

Yong Li , Student Member, IEEE,JiabeiZeng , Member, IEEE, Shiguang Shan , Member, IEEE,

and Xilin Chen, Fellow, IEEE

Each representation is weighed via a proposed gate unit that

computes an adaptive weight from the region itself according

to the unobstructedness and importance. Considering different

RoIs, we introduce two versions of ACNN: patch-based ACNN

(pACNN) and global–local-based ACNN (gACNN). pACNN only

pays attention to local facial patches. gACNN integrates local rep-

resentations at patch-level with global representation at image-

dataset with real-world occlusions, the two largest in-the-wild

occluded patches to other related but unobstructed ones. ACNNs

also outperform other state-of-the-art methods on several widely

used in-the-lab facial expression datasets under the cross-dataset

evaluation protocol.

Index Terms— Facial expression recognition, occlusion, CNN

with attention mechanism, gate unit.

Manuscript received May 28, 2018; re vised September 27, 2018 and

November 11, 2018; accepted December 5, 2018. Date of publication

December 14, 2018; date of current version February 13, 2019. This

work was partially supported by National Key R&D Program of China

Academy of Sciences, Beijing 100049, China (e-mail: yong.li@vipl.ict.ac.cn;

xlchen@ict.ac.cn).

J. Zeng is with the Key Laboratory of Intelligent Information Processing,

S. Shan is with the Key Laboratory of Intelligent Information Processing,

Center for Excellence in Brain Science and Intelligence Technology, Institute

I. INTRODUCTION

over r ecent decades, as it holds promise to an abundance

of applications, such as human-computer interaction, affect

able to perform poorly when recognizing human expressions

faces. It is non-trivial to address the occlusion issue because

occlusions varies in the occluders and their positions. The

occlusions may caused by hair, glasses, scarf, breathing mask,

hands, arms, food, and other objects that could be placed in

front of the faces in daily life. These objects may block the eye,

variability of occlusions cannot be fully covered by limited

cheek), or other highly related facial regions (e.g., regions

automatically perceives the blocked facial patches and pays

Fig. 1 illustrates the main idea of the proposed method.

the last three visualized patches are blocked by the baby’s

2440 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 28, NO. 5, MAY 2019

region, a specific Patch-Gated Unit (PG-Unit) is learnt to weigh the local representations according to the region’s “unobstructed-ness” (to what extent the

patch is occluded). Then, the weighed representations are concatenated and passed to the classification part. gACNN integrates weighed local representations

with global representation (purple rectangle). The global representation is encoded and weighed via a Global-Gated Unit (GG-Unit).

Considering different facial regions of interest, we propose

two versions of ACNN: (1) pACNN crops patches of interest

from the last convolution feature maps according to the

positions of the related facial landmarks. Then for each patch,

paper, we provide technical details of facial region decompo-

sition, present extended results with more comparisons and on

tion mechanism (ACNN) to recognize facial expressions

perceives the occluded regions of the face and focus on

the most informative and un-blocked regions.

ment on FER performance under occlusions.

3) Experimental results demonstrate the advantages of the

either partially occluded or non-occluded faces.

the presence of real occlusions (FED-RO). To the best of

the p resence of real occlusions.