essentially composed of multiple channels: hand
location, handshape, hand orientation, hand movement,
eye gazing, head tilting, shoulder tilting, body gesture,
and facial expression. Linguistic meaning is described
and represented by these information channels in sign
language. This multi-channel nature of CSL results in the
difficulty of coding sign languages to linear single-
channel character string. Foreign scholars believe that
sign languages also have writing systems, such as the
SignWriting system [6], ASL-phabet [7], and
HamNoSys [8]. However, these writing systems have
narrow range of users. The report sent by SignWriting
(sw-l@majordomo.valenciacc.edu) in groups shows that
at least 14 schools all over the world are currently using
this system. The narrow range of users explains why the
deaf community is small on the scale and can be
shrinking because of population, politics, technology,
and other factors [9].
Even if CSL has a writing system, its multi-channel
nature will definitely result in the system losing many
linguistic details. The most ideal understanding of sign
languages should be that the visual characteristics of the
visual-spatial language in sign language directly reach
the semantic units in the brain rather than initially
transforming it to written texts and then to semantics.
This approach is the most natural method of
comprehending sign languages in the brain. From this
perspective, this paper presents a computational
cognitive model for CSL comprehension on the
cognitive functionalities in the human brain combined
with a knowledge representation theory of artificial
intelligence. Therefore, we think that an ideal model is
quantitative in a programmable way.
The rest of this paper is organized as follows: First,
the background of CSL understanding is presented to
provide a brief introduction of the problems related to
CSL information processing. Second, we present a
systemic review on psychological and
neurophysiological studies with converging evidences to
uncover the cognitive and neural mechanisms of sign
language comprehension in the human brain. Third, a
computational model for CSL comprehension based on
the cognitive mechanism of sign language is proposed.
Fourth, the relevant meanings of a sign are considered as
nodes within a semantic neural network, and the
relevance between each meaning and corresponding sign
is regulated using the spreading activation theory.
Finally, the last section is the conclusion and future work.
2 SIGN LANGUAGE PERCEPTION AND COMPREHENSION
IN THE HUMAN BRAIN
An important source of inspiration in the research on
natural language processing is the cognitive brain
mechanism. The human brain has abundant sensing
organs and can thus abstract the overall knowledge of
languages from the perceived information and then
complete the understanding languages, thereby realizing
more complex intellectual activities. The brain delivers
and exchanges 1 PB data one trillion times every second,
as well as processes sound, sign, image, and other data
synchronously. The human brain is clearly an inborn
natural language processor [10].
Exploring the computational cognitive models on
how human brains comprehend sign language in both
areas of computational linguistics and cognitive
computation is much desired. This section summarizes
the psychological and neurophysiological findings for
sign language perception and comprehension in the
human brain. A summary follows the cognitive process
[11], including the mental processes of perception,
memory, and judgment. A computational cognitive
model is developed in Section 3 on the basis of these
findings.
A. Perception
Sign language exploits visual-spatial mechanisms to
express grammatical structure and function. Visual-
spatial perception, memory, and mental transformations
are prerequisites to grammatical processing in American
Sign Language [12], and are also central to visual mental
imagery [13].
A series of experiments are conducted to investigate
visual attention [14]. Movement recognition in the
peripheral vision is important in sign perception because
the signers mainly look at the face instead of keeping
track of the hands when their communicating through
sign language [15]. Therefore, lexical identification
depends on peripheral vision when signs are produced
away from the face. The recognition of movement
direction seems to be the selective functions of peripheral
vision [16]. At present, whether deaf subjects only have
strong perceiving ability of the peripheral vision or are
more efficient in allocating attention to peripheral vision
remains unclear. Literature [17] showed that auditory
deprivation can change visual attention processing. They
determined that deaf subjects did not shift their attention
when processing the information (alphabet set) presented
in central vision, while hearing subjects had to shift their
attention to search for the alphabet set continuously.
Similarly, Literature [18] also determined that the lack of
auditory input would cause weak selective (or more
distributed) visual attention among deaf children.
Literature [17] proposed that intermodal sensory
compensation results in more effective visual processing;
that is, the strong allocation of visual attention can be
attributed to neuron reorganization caused by auditory
deprivation from birth. Recent MRI evidence supports
this hypothesis [19].
These findings are selective attention cases, where
attention selectively processes certain stimuli but ignores
other stimuli. The cases refer to the selective orientation
and concentration of people’s senses (i.e., visual,
auditory, and tastable sense) and consciousness (i.e.,
awareness and thinking) on certain targets synchronously
(will toward other factors). Studies on attention cannot