分布式网络_分布式网络资源-CSDN文库

需积分: 9 9 浏览量 2018-06-18 10:05:47 上传评论收藏 772KB PDF 举报

分布式网络是计算机网络的一种形式，其中多个计算机通过网络互联，共同完成一个任务，而不是所有计算任务都集中在单一的中央服务器上。这种架构的出现主要是为了解决大规模数据处理和计算任务所面临的性能瓶颈问题。分布式网络技术在各种应用场景中发挥着重要作用，例如云计算、大数据处理、以及近年来兴起的人工智能和机器学习领域。在分布式网络环境下，神经网络训练可以显著减少时间和内存成本。神经网络是人工智能和机器学习中非常重要的一个算法类别，尤其在图像识别、语音识别、自然语言处理等领域有着广泛的应用。然而，传统上，训练大型神经网络需要大量的计算资源和时间，尤其当面对海量数据时。为了克服这些问题，分布式训练方法应运而生。文章中提到的Random Vector Functional-Link (RVFL)网络是一种特殊的前馈神经网络，其中输入层和输出层之间至少有一个隐藏层。RVFL网络的特点是隐藏层的参数在训练过程中是固定的，这样可以极大地提高训练的速度和降低复杂性。RVFL网络在很多实际问题中展现了良好的性能，比如函数逼近、模式识别、时间序列预测等。为了使RVFL网络的训练过程适合于分布式环境，文章提出了两种基于分布式平均一致性（Decentralized Average Consensus，DAC）和交替方向乘子法（Alternating Direction Method of Multipliers，ADMM）策略的分布式学习算法。DAC和ADMM是分布式优化领域中的重要方法，能够使网络中的不同节点在没有中央控制机构协调的情况下，通过局部通信和计算达到全局的一致性。分布式学习的目标是构建一个共有的学习模型，从而优化整个系统在所有局部数据集上的性能。文章假设所有站点知道输入层的初始权重，而局部RVFL网络的输出权重只能通过相邻节点的通信渠道进行共享。在严格的局部数据集划分下，提出的算法能够在分布式环境中有效运行。通过在五个基准数据集上对所提出的算法进行评估，实验结果显示，基于DAC的学习算法在效果、效率和计算复杂度方面表现较好，而基于ADMM的学习算法虽然具有较高的准确度，但计算负担较重。分布式学习算法的提出和应用，不仅解决了大规模数据集在单个计算节点上训练时面临的问题，也为大规模深度学习模型的快速迭代提供了可能。这种分布式训练方法可以使多个计算节点协同工作，将数据分割成小部分，各节点独立地进行计算任务，然后通过某种形式的协调机制来汇总计算结果，从而减少训练时间并提高训练效率。此外，分布式方法还有助于提供容错能力，即使某些节点出现故障，其他节点仍然可以继续工作，不会影响整个模型的训练过程。

资源推荐

资源详情

资源评论

Distributed learning for Random Vector Functional-Link

networks

Simone Scardapane

⇑

, Dianhui Wang

, Massimo Panella

, Aurelio Uncini

Department of Information Engineering, Electronics and Telecommunications (DIET), ‘‘Sapienza’’ University of Rome, Via Eudossiana 18, 00184 Rome, Italy

Department of Computer Science and Computer Engineering, La Trobe University, Melbourne, VIC 3086, Australia

article info

Article history:

Received 28 October 2014

Received in revised form 9 December 2014

Accepted 8 January 2015

Available online 13 January 2015

Keywords:

Random Vector Functional-Link

Distributed learning

Consensus

Distributed Optimization

abstract

This paper aims to develop distributed learning algorithms for Random Vector Functional-

Link (RVFL) networks, where training data is distributed under a decentralized information

structure. Two algorithms are proposed by using Decentralized Average Consensus (DAC)

and Alternating Direction Method of Multipliers (ADMM) strategies, respectively. These

algorithms work in a fully distributed fashion and have no requirement on coordination

from a central agent during the learning process. For distributed learning, the goal is to

build a common learner model which optimizes the system performance over the whole

set of local data. In this work, it is assumed that all stations know the initial weights of

the input layer, the output weights of local RVFL networks can be shared through commu-

nication channels among neighboring nodes only, and local datasets are blocked strictly.

The proposed learning algorithms are evaluated over ﬁve benchmark datasets. Experimen-

tal results with comparisons show that the DAC-based learning algorithm performs favor-

ably in terms of effectiveness, efﬁciency and computational complexity, followed by the

ADMM-based learning algorithm with promising accuracy but higher computational

burden.

1. Introduction

Over the past decades, supervised learning techniques have been well developed with theoretical analyses and empirical

studies [15]. The ICT world, however, is being rapidly reshaped by emerging trends such as big data [22], pervasive comput-

ing [33], commodity computing [11], Internet of things [2], and several others. All these frameworks have a similar common

theme underlying them: computing power is now a widespread feature surrounding us, and the same can be said about data.

Consequently, supervised learning is expected to face major technological and theoretical challenges, since in many situa-

tions the overall training data cannot be assumed to lie at a single location, nor is it realistic to have a centralized authority

for collecting and processing it. The previous trends also put forth the challenge of analyzing structured and heterogeneous

data [4], however, we are not concerned with this issue in this paper.

As a prototypical example, consider solving a music classiﬁcation task (e.g., genre classiﬁcation [35]) over a peer-to-peer

(P2P) network of computers, each node possessing its own labeled database of songs. It is reasonable to assume that, to

obtain good performance, no single database may be sufﬁcient, and there is the need of leveraging over the data of all users.

http://dx.doi.org/10.1016/j.ins.2015.01.007

⇑

Corresponding author. Tel.: +39 06 44585495; fax: +39 06 4873300.

E-mail addresses: simone.scardapane@uniroma1.it (S. Scardapane), dh.wang@latrobe.edu.au (D. Wang), massimo.panella@uniroma1.it (M. Panella),

aurelio.uncini@uniroma1.it (A. Uncini).

Information Sciences 301 (2015) 271–284

Contents lists available at ScienceDirect

Information Sciences

journal homepage: www.elsevier.com/locate/ins

However, in a P2P network no centralized authority exists, hence the nodes need a distributed training protocol to solve the

classiﬁcation task. In fact, it is known that a fully decentralized training algorithm can be useful even in situations where

having a master node is technologically feasible [16]. In particular, such a distributed algorithm would remove the risks

of having a single point of failure, or a communication bottleneck towards the central node. Similar situations are also wide-

spread in Wireless Sensor Networks (WSN), where additional power concerns arise [5]. Finally, it may happen that data sim-

ply cannot be moved across the network: either for being large (in term of number of examples or dimensionality of each

pattern), or because fundamental privacy concerns are present [40]. The general setting, which we will call ‘data-distributed

learning’, is graphically depicted in Fig. 1.

So far, a large body of research has gone into developing fully distributed, decentralized learning algorithms, including

works on diffusion adaptation [21,34], learning by consensus [16], distributed learning on commodity clusters architectures

[8], adaptation on WSNs [5,32], distributed online learning [13], distributed optimization [6,14,18,38], ad-hoc learning algo-

rithms for speciﬁc architectures [12,26], distributed databases [20], and others. Despite this, many important research ques-

tions remain open [31], and in particular several well-known learning models, originally formulated in the centralized

setting, have not yet been generalized to the fully decentralized setting.

In this paper, we propose two distributed learning algorithms for a yet-unexplored model, that is Random Vector Func-

tional-Link (RVFL) networks [1,10,30,37]. As illustrated successively, RVFLs can be viewed as feedforward neural networks

with a single hidden layer, resulting in a linear combination of a (ﬁxed) number of non-linear expansions of the original

input. A remarkable characteristics of such a learner model lies in the way of parameter assignment, that is, the input

weights and biases are randomly chosen and ﬁxed in advance before training. Despite this simpliﬁcation, RVFLs can be

shown to possess universal approximation capabilities, provided a sufﬁciently large set of expansions [17]. This grant them

with a number of peculiar characteristics, making them particularly suited in a distributed environment. In particular, RVFL

models are linear in the parameters, thus optimal parameters can be found with a standard linear regression routine, which

can be implemented efﬁciently even in low-cost hardware, such as sensors or mobile devices [30]. In fact, the optimum of the

training problem can be formulated in a closed form, involving only matrix inversions and multiplications, making the model

efﬁcient even when confronted with large amounts of data. Finally, the same formulation can be used equivalently in the

classiﬁcation and in the regression setting. In this paper, we focus on batch learning scheme development, however the pro-

posed algorithms can be further extended for sequential learning with the use of standard gradient-descent procedures [10],

whose decentralized formulation have been only partially investigated in the literature [34].

The key idea behind the proposed algorithms is to let all nodes train a local model (simultaneously) using the subset of

training data, followed by ﬁnding the common output weights of the master learner model. Two effective approaches for

deﬁning the common output weights are adopted in this study. One is the Decentralized Average Consensus (DAC) strategy

[28], and another is the well-known Alternating Direction Method of Multipliers (ADMM) algorithm [6]. DAC is an efﬁcient

protocol to compute averages over very general networks, with two main characteristics. Firstly, it does not require a cen-

tralized authority coordinating the overall process, and secondly, it can be easily implemented even on the most simple net-

works [16]. These characteristics have made DAC an attractive method in many distributed learning algorithms, particularly

in the ‘learning by consensus’ theory outlined in [16]. From a theoretical viewpoint, the DAC-based algorithm is similar to a

bagged ensemble of linear predictors [7], and despite its simplicity and non-optimal nature, our experimental simulations

show that it results in highly competitive performance. The second strategy (ADMM) is the most widely employed distrib-

uted optimization algorithm in machine learning (e.g. for LASSO [6] and Support Vector Machines [14]), making it a natural

candidate for the current research. This second strategy is more computational demanding than the DAC-based one, but it

has high theoretical guarantees in term of convergence, speed and accuracy. Our simulation results obtained from both algo-

rithms are quite promising and comparable to a centralized model exploiting the overall dataset. Moreover, the consensus

strategy is extremely competitive on a large number of realistic network topologies.

The remainder of the paper is organized as follows. Section 2 brieﬂy reviews RVFLs and its learning algorithm, and intro-

duces the DAC algorithm. Section 3 describes the data-distributed learning framework, and proposes two training algorithms

for RVFLs models. Sections 4 and 5 detail the experimental setup and the numerical results on four realistic datasets, plus an

additional experiment on a large-scale image classiﬁcation task, respectively. Section 6 concludes this paper with some dis-

cussions and future possible researches.

Fig. 1. Supervised learning in a network of agents: data is distributed throughout the nodes, and all of them must converge to a single learning model. For

readability, we assume undirected connections between agents.

272 S. Scardapane et al. / Information Sciences 301 (2015) 271–284

2. Preliminaries

This section provides some supportive results that will be used in the subsequent sections. We start from formulating

some basic concepts related to RVFL networks with a least-square solution as its learning algorithm (Section 2.1). Then,

we brieﬂy introduce the DAC algorithm, for evaluating global averages under a decentralized information structure

(Section 2.2).

2.1. Random-vector functional links

Let us consider ﬁrst a regression problem with one-dimensional scalar outputs y 2 R. A Functional Link Artiﬁcial Neural

Network (FLANN) with a single output neuron can be described as a weighted sum of B non-linear transformations of the

input [30]:

f ðxÞ¼

m¼1

ðx; w

Þ¼b

hðx; w

; ...; w

Þ; ð1Þ

where the mth transformation is parametrized by the vector w

, and the input is a d-dimensional real vector x 2 R

. Each

h : X ! R is called a base, hidden function, or functional link. In particular, in our simulations we will use sigmoid basis

functions given by:

hðx; w; bÞ¼

1 þ exp w

x þ bfg

: ð2Þ

The parameters w

; ...; w

are chosen in the beginning of the learning process, independently of the training data. In partic-

ular, in a RVFL neural network the parameters of the hidden functions are chosen randomly from a predeﬁned probability

distribution [1,17,37]. A schematic depiction of a RVFL with two inputs, three hidden nodes, and one output is given in Fig. 2.

Under moderate assumptions on the smoothness of the underlying function, universal approximation capability of RVFLs

is guaranteed provided a sufﬁciently large number of hidden functions, i.e., provided that B is large enough [17]. In this way,

the problem of learning a model of the form in (1) is reduced to a linear regression over the coefﬁcients b ¼ b

; ...; b

½

. Sup-

pose we are provided with a set of N samples of the desired function, called the training set, that is S ¼ x

; y

fg; i ¼ 1 ...N.

Denote by H the hidden matrix:

H ¼

ðx

Þh

ðx

Þh

ðx

; ð3Þ

where we have dropped the parametrization with respect to the w

parameters for readability. Choosing the optimal b can

be formulated as a standard regularized least-square problem:



¼ arg min

b2R

kHb  Yk

; ð4Þ

where Y is a column vector whose ith element is the output sample y

associated with the pattern x

; i ¼ 1 ...N, of the train-

ing set. The problem in (4) is strictly convex, so its solution can be found by setting the gradient of JðbÞ to 0:

¼ H

Hb  H

Y þ kb ¼ 0; ð5Þ

from which we obtain the well-known solution:



¼ H

H þ kI



1

Y; ð6Þ

Fig. 2. Schematic depiction of a RVFL architecture with two inputs, three hidden nodes, and one output. Fixed connections are shown as dashed lines, while

trainable connections as ﬁxed lines.

S. Scardapane et al. / Information Sciences 301 (2015) 271–284

273