One key challenging issue of facial expression recognition
is to capture the dynamic variation of facial physical
structure from videos. In this paper, we propose a part-based
hierarchical bidirectional recurrent neural network (PHRNN) to
analyze the facial expression information of temporal sequences.
Our PHRNN models facial morphological variations and dynamical
evolution of expressions, which is effective to extract “temporal
features” based on facial landmarks (geometry information)
from consecutive frames. Meanwhile, in order to complement the
still appearance information, a multi-signal convolutional neural
network (MSCNN) is proposed to extract “spatial features” from
still frames. We use both recognition and verification signals
as supervision to calculate different loss functions, which are
helpful to increase the variations of different expressions and
reduce the differences among identical expressions. This deep
evolutional spatial-temporal network (composed of PHRNN and
MSCNN) extracts the partial-whole, geometry-appearance, and
dynamic-still information, effectively boosting the performance of
facial expression recognition. Experimental results show that this
method largely outperforms the state-of-the-art ones. On three
widely used facial expression databases (CK+, Oulu-CASIA, and
MMI), our method reduces the error rates of the previous best
ones by 45.5%, 25.8%, and 24.4%, respectively
