ConvergenceofgradientmethodwithpenaltyforRidgePolynomialneuralnetwork资源-CSDN文库

研究论文

156 浏览量 2021-02-21 04:57:40 上传评论收藏 504KB PDF 举报

资源推荐

资源详情

资源评论

Letters

Convergence of gradient method with penalty for Ridge Polynomial

neural network

Xin Yu

, Qingfeng Chen

School of Computer, Electronics and Information, Guangxi University, Nanning 53004, China

article info

Article history:

Received 2 December 2011

Received in revised form

24 April 2012

Accepted 28 May 2012

Communicated by R.W. Newcomb

Available online 3 July 2012

Keywords:

Ridge Polynomial neural network

Gradient algorithm

Monotonicity

Convergence

abstract

In this paper, a penalty term is added to the conventional error function to improve the generalization

of the Ridge Polynomial neural network. In order to choose appropriate learning parameters, we

propose a monotonicity theorem and two convergence theorems including a weak convergence and a

strong convergence for the synchronous gradient method with penalty for the neural network. The

experimental results of the function approximation problem illustrate the above theoretical results

are valid.

1. Introduction

The pi–sigma(

–

) neural network (PSNN) [1] isatypeof

high-order feedforword neural network, which uses t he product of

the sums of the input units. This network has a powerful function

approximation ability while avoiding the combinatorial explosion of

the higher terms. However, it is not an universal approximator. The

Ridge Polynomial neural network (RPNN) [2] is a generalization of

PSNN and uses a number of PSNNs as its basic building blocks. It

maintains the fast learning ability of the PSNN and can uniformly

approximate any continuous function on a compact set in R

[3].

Thus, the RPNN has been widely used in various ﬁelds [4–6].

The synchronous gradient method is a commonly used train-

ing algorithm for higher-order neural networks, and some con-

vergence theorems of gradient methods for various neural

networks have been proposed [7–13]. The study on the conver-

gence of training method is important for choosing appropriate

training parameters, e.g., initial weights and learning rate, to

perform an effective network training. Moreover, the general-

ization of neural networks is a critical issue in designing a training

method for neural networks. To improve the generalization of

neural networks, one widely used scheme is adding a penalty

term proportional to the magnitude of network weights to the

conventional error function [10–13]. Thus, this article aims to

study the convergence of synchronous gradient method with

penalty for the RPNN. We ﬁrst prove that the error sequence will

be uniformly monotonous and the algorithm also will be weakly

convergent during the training procedure on condition that

certain assumptions (cf. Assumptions A1–A3) are satisﬁed, then

prove that if the error function has ﬁnite stable points, the

algorithm with the weak convergence will be strongly conver-

gent. In the end, the training method is applied to a function

approximation problem to illustrate our theoretical ﬁnding.

2. Preliminary

Since the Ridge Polynomial neural network mainly consists of

some Pi–Sigma neural networks, the structure of the Pi–Sigma

neural network is introduced below.

2.1. Pi–sigma neural network

Fig. 1 shows a Pi–Sigma neural network with a single output.

Let nþ1, k and 1 be the number of the input units, summing units

and output units of the PSNN, respectively. Denote by w

¼ðw

, ..., w

A R

n þ1

the weight vector from the summing unit j to

the input units, and write w ¼ðw

, w

, ..., w

A R

ðn þ1Þk

. Note that

the weights from summing units to product unit are ﬁxed to 1. Let

x ¼ðx

, x

, ..., x

A R

n þ1

be nþ1 dimensional input vector where

¼1 is an adjustable threshold. Further, let y denote the output

of the product unit, given by

y ¼

j ¼ 1

i ¼ 0

j ¼ 1

, x



ð1Þ

Contents lists available at SciVerse ScienceDirect

journal homepage: www.elsevier.com/locate/neucom

Neurocomputing

http://dx.doi.org/10.1016/j.neucom.2012.05.022

Corresponding author.

E-mail addresses: yuxin21@126.com (X. Yu), qingfeng@gxu.edu.cn (Q. Chen).

Neurocomputing 97 (2012) 405–409

本内容试读结束，登录后可阅读更多

下载后可阅读完整内容，剩余4页未读，立即下载

评论收藏

内容反馈

weixin_38565631

粉丝: 2
资源: 913

Convergence of gradient method with penalty for Ridge Polynomial...

最新资源

Convergence of gradient method with penalty for Ridge Polynomial...

Convergence

A Self-Adjusting Conjugate Gradient Method with Sufficient Descent Condition .

Wind Prediction Based on Improved BP Artificial Neural Network in Wind Farm

feed-forward neural network Vs. CNN

Training Neural Networks without Gradients

Information-Theoretic Aspects of Neural Networks

Convergence of CR-gradient learning algorithm for Multi-valued Neuron

AdaBatch: Efficient Gradient Aggregation.pdf

The convergence analysis of Carroll function method for nonlinear SOC programming

On convergence rate of distributed stochastic gradient algorithm for convex optimization with inequality constraints

Expanding the Range of Convergence of the CORDIC Algorithm

现代数字通信论文1

Convergence properties of the nelder-mead simplex method in low dimens

On the convergence of an adaptive finite element methodfor a second-kind elliptic variational inequality

Polynomial Convergence of An Inexact Infeasible Interior Point Algorithm For P-Matrix Linear Complementarity Problems

On Local and Global Convergence of a Nonsmooth Newton-type Method for Nonlinear Semidefinite Programs

Convergence analysis of an adaptive finite element method for the elasto-plastic torsion problem

Real-Time Accurate Mapping of Large Environments

On the convergence of hybrid polynomial approximation to higher derivatives of rational curves

最新资源