However, nearly none of them pay attention to the underlying rela-
tionship between the manifold distribution and given classes, thus
unable to discover the knowledge concealed in data. As a result, an
open and challenging problem is to design a framework for mani-
fold data with the goal of combining the advantages of clustering
and classification and meanwhile revealing the statistical relation-
ship between manifolds and classes.
In this paper, we propose a manifold learning framework for
both clustering and classification (MCC). MCC aims to discover
the manifold structure hidden in data, design an effective and
transparent classification mechanism and meanwhile exploit the
relationship between manifolds and classes. To achieve these
goals, our framework treats the manifold clustering learning and
classification learning in a two-step sequential manner. In the first
step, the clustering through ranking on manifolds is performed to
explore structures in data; in the second step, by using the Baye-
sian rule, the class posterior probability is calculated to give class
labels for unseen samples. It is worth mentioning that the number
of manifolds (i.e. clusters) has a significant influence on the result
of manifold clustering [25–27]. To auto-determine this parameter
in our algorithm, the inter-cluster mean distance by ranking on
manifolds is maximized and while the intra-cluster mean distance
is minimized. As a result, our algorithm can auto-determine the
clustering parameters without manual determining. Another key
of this framework is to connect the multi-manifold with the given
classes employ, and then establish a relationship between them.
This relationship creates a bridge between clustering learning
and classification learning. Based on such relationship, our frame-
work cannot only group multi-manifold into different clusters, but
also make classification decisions for unseen samples. More impor-
tantly, this relationship can successfully reflect the probability and
statistics meaning between manifold structures and given classes,
so that we gain some meaningful insights to make MCC prone to be
transparent.
The new manifold learning framework for both clustering and
classification is interesting from a number of perspectives:
(1) Our algorithm can perform manifold clustering learning
which can auto-determine the clustering parameters with-
out manual determining.
(2) Our algorithm can perform manifold classification learning
which models the posterior probabilities pð
x
l
jx
i
Þ by using
the Bayesian rule.
(3) Our algorithm can provide the statistical relationship
between the manifold structure and the given classes.
The experimental results on both synthetic and real-life data-
sets all demonstrate the effectiveness and potential of MCC.
The rest of this paper is organized as follows: Section 2 reviews
the related works. Section 3 describes the proposed manifold
learning framework for both clustering and classification. Prelimi-
nary experimental results are shown in Section 4. Finally, we give
concluding remarks and future work in Section 5.
2. Related works
There have been several recent related works to inherit the
merits of both clustering and classification learning. We review
the main works as follows.
2.1. Fuzzy relational classifier
Fuzzy Relational Classifier (FRC) [12] was proposed to provide a
transparent alternative to the black-box techniques such as neural
networks. As show in Fig. 1, in FRC, FCM is firstly adopted as the
clustering criterion to discover the natural structure in data, and
its objective function is as follows:
J
FCM
ðU; VÞ¼
X
c
j¼1
X
n
i¼1
u
2
ji
x
i
v
j
2
; ð1Þ
where fx
1
; x
2
; ...; x
n
g and f
v
1
;
v
2
; ...;
v
c
g are the training samples
and cluster centers, respectively; and u
ji
is the fuzzy memberships
of x
i
to
v
j
. By definition, each sample x
i
satisfies the constraint
P
c
j¼1
u
ji
¼ 1. And then, a relation matrix R is computed for the
obtained fuzzy partition and the given hard class labels. In FRC,
FCM is unable to group the datasets consisting of the non-
spherical clusters, so that the interpretation of the clustering or
classification results may be biased.
Afterwards, we have presented Robust FRC (RFRC) [13] to
improve both clustering and classification performance of FRC in
our previous work. Specifically, in the clustering phase, the robust
Kernelized FCM (KFCM) [14,15] is adopted to replace FCM which
can be described as below:
J
KFCM
ðU; VÞ¼
X
c
j¼1
X
n
i¼1
u
m
ji
/ðx
i
Þ/ð
v
j
Þ
2
; ð2Þ
where / is an implicit nonlinear map from the input space to a
rather high dimensional feature space. Compared to FCM, KFCM
based on RBF kernel is a robust estimator according to
M-estimator and is more flexible for clustering non-spherical data.
Next, in the classification phase, the soft class label motivated by
the fuzzy k-nearest-neighbor [28] is employed to replace the hard
class label. With the incorporation of both KFCM and the soft class
labels, RFRC makes the constructed relation matrix R more really
reflect the relationship between the classes and clusters, and thus
significantly boosts the performance of FRC.
It is worth to point out that in FRC and RFRC, the entries in the
relation matrix R lack the statistical meaning, thus it is difficult to
judge whether the obtained relationship is really reliable.
2.2. Radial basis function neural networks
Radial Basis Function neural networks (RBFNN) [16,17],as
shown in Fig. 2, is a feed-forward multi-layer network. It usually
consists of three layers: input layer, hidden layer and output layer.
Each basis function
U
k
corresponds to a hidden unit and w
kl
repre-
sents the weight from the kth basis function or hidden unit to the
lth output units.
In the training phase of RBFNN, the basis function
U
k
for each
hidden node can be determined by
U
RBF
k
x;
v
k
ðÞ¼exp
x
v
k
kk
2
2
r
2
!
; ð3Þ
Training Data
Exploratory Data
Analysis
Logical
Interpretation
Features
Unsupervised
Fuzzy Clustering
Cluster Means
Fuzzy
Partition
Class Labels
-composition,
aggregation
Fuzzy Relation
Fig. 1. Training process of FRC and RFRC.
642 W. Cai / Knowledge-Based Systems 89 (2015) 641–653