【免费】SpikingNeuralNetworksandBio-InspiredSupervisedDeepLearnin资源-CSDN文库

人工智能

需积分: 0 116 浏览量 2023-08-02 11:28:27 上传评论收藏 1.04MB PDF 举报

资源推荐

资源详情

资源评论

Spiking Neural Networks and Bio-Inspired Supervised Deep Learning: A Survey

GABRIELE LAGANI

∗

, FABRIZIO FALCHI, CLAUDIO GENNARO, and GIUSEPPE AMATO, ISTI-CNR,

Italy

For a long time, biology and neuroscience elds have been a great source of inspiration for computer scientists, towards the development

of Articial Intelligence (AI) technologies. This survey aims at providing a comprehensive review of recent biologically-inspired

approaches for AI. After introducing the main principles of computation and synaptic plasticity in biological neurons, we provide a

thorough presentation of Spiking Neural Network (SNN) models, and we highlight the main challenges related to SNN training, where

traditional backprop-based optimization is not directly applicable. Therefore, we discuss recent bio-inspired training methods, which

pose themselves as alternatives to backprop, both for traditional and spiking networks. Bio-Inspired Deep Learning (BIDL) approaches

towards advancing the computational capabilities and biological plausibility of current models.

CCS Concepts: • Computing methodologies → Bio-inspired approaches; Bio-inspired approaches.

Additional Key Words and Phrases: Bio-Inspired, Hebbian, Deep Learning, Neural Networks, Spiking

ACM Reference Format:

Gabriele Lagani, Fabrizio Falchi, Claudio Gennaro, and Giuseppe Amato. 2018. Spiking Neural Networks and Bio-Inspired Supervised

Deep Learning: A Survey. 1, 1 (August 2018), 31 pages. https://doi.org/XXXXXXX.XXXXXXX

1 INTRODUCTION

Spiking Neural Networks (SNN) have recently emerged as a biologically inspired alternative to traditional Deep Learning

(DL) models, towards addressing the current limitations of Deep Neural Networks (DNNs) in terms of ecological impact

[

]. Indeed, biological brains exhibit extraordinary capabilities in terms of energy eciency, supporting advanced

cognitive functions while consuming only 20W [

]. It is believed that the key to the energy ecient computation

of biological neurons lies in the particular coding paradigm based on short pulses, or spikes [

]. SNN models aim at

simulating the behavior of biological neurons more realistically, compared to traditional DNNs. As a result, SNNs are

well suited for energy-ecient implementations in neuromorphic [

174

186

190

229

] or biological [

111

176

]

hardware. This makes SNNs a promising direction toward energy-ecient DL.

Unfortunately, training SNNs is not trivial, as traditional optimization based on the backpropagation algorithm

(backprop) is not directly applicable [

165

]. In fact, the biological plausibility of backprop – the workhorse of DL – is

questioned by neuroscientists [

113

130

157

172

]. Therefore, researchers took again inspiration from biology, in

order to nd new learning solutions as alternatives to backprop. The goal was not only to address the problem of

SNN training [

148

], but also to discover novel approaches to the learning problem [

139

182

], and possibly more

data ecient strategies [

105

–

107

110

]. In fact, another limitation of current DL solutions is the requirement of

∗

Corresp.

Authors’ address: Gabriele Lagani, gabriele.lagani@isti.cnr.it; Fabrizio Falchi, fabrizio.falchi@isti.cnr.it; Claudio Gennaro, claudio.gennaro@isti.cnr.it;

Giuseppe Amato, giuseppe.amato@isti.cnr.it, ISTI-CNR, Pisa, Italy, 56124.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not

made or distributed for prot or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components

of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to

redistribute to lists, requires prior specic permission and/or a fee. Request permissions from permissions@acm.org.

Manuscript submitted to ACM

Manuscript submitted to ACM 1

arXiv:2307.16235v1 [cs.NE] 30 Jul 2023

2 Lagani, et al.

Fig. 1. A schematic view of the topics of Bio-Inspired Deep Learning (BIDL) addressed in this work. We discuss SNN models, which

aim at providing a biologically faithful simulation of real neurons, and they find applications in the context of energy-eicient

biological and neuromorphic computing. Learning strategies for SNNs include Spike Time Dependent Plasticity (STDP), surrogate

gradient strategies for adapting backprop optimization to the case of SNNs, etc. We also discuss training algorithms that pose

themselves as alternatives to backprop. Some approaches aim at approximating backprop using only biologically plausible synaptic

updates, while others, rather than simply approximating backprop, consider dierent perspectives, such as forward propagation

approaches, reservoir computing, etc.

large amounts of training data. On the other hand, biological brains also show interesting properties in terms of data

eciency, being able to infer new knowledge and easily generalize from little experience [112].

In this survey, we illustrate biologically realistic SNN models of neural computation, and we discuss spike coding

strategies, spiking models of synaptic plasticity, and the training challenges and application potentials of this line of

research. We also consider the learning problem in more detail, illustrating recently proposed methodologies for SNN

and DNN training, which pose themselves as alternatives to traditional backprop-based training.

One of the goals of this survey is to provide a comprehensive review of the methods coming from biological

inspiration, and their potential impact on current DL technologies. The eld of Bio-Inspired Deep Learning (BIDL)

observes the convergence of a broad spectrum of ideas, coming from the computer science and neuroscience elds;

hence, one of the goals of this work is to highlight the connections between these two viewpoints. Fig. 1 provides a

schematic visualization of the topics addressed in this work.

This document can be of interest both to novel readers who approach themes in the BIDL eld for the rst time,

as well as a reference for researchers already familiar with these topics. Moreover, this document does not require

prerequisite knowledge in the neuroscience domain, but the necessary background is provided where needed. At the

same time, neuroscientists that are curious about the engineering aspects behind AI architectures could also nd this

document an interesting resource.

Manuscript submitted to ACM

Bio-Inspired Deep Learning 3

This survey is organized as follows:

• Section 2 discusses related surveys in the BIDL eld.

• Section 3 briey introduces some background about biological neurons and synaptic plasticity models.

•

Section 4 discusses biologically detailed models of neural computation based on SNNs, highlights the challenges

of training such models, but also their technological potential for energy-ecient biological and neuromorphic

computing.

• Section 5 explores alternative DNN training algorithms which do not require backprop.

•

Finally, we present our conclusions in Section 6, outlining open challenges and possible future research directions.

Moreover, in our companion paper [

] we discuss biological models of synaptic plasticity in greater detail, and we

highlight the relationships between such models and unsupervised pattern discovery mechanisms in neural networks,

showing the connections between neuroscientic principles and emerging cognitive behavior.

2 RELATED SURVEYS

The eld of BIDL has been the subject of increasing attention recently, and it has been reviewed in a number of recent

contributions from dierent perspectives. In particular, there are two perspectives from which the eld is approached:

the neuroscience and the computer science/engineering viewpoints. Related contributions so far have been more tied to

one of these viewpoints, or they have tackled specic aspects of this eld. In this perspective, the goal of our contribution

is to provide a broader perspective on the various concepts that emerge in the eld of BIDL, and to highlight the

connections between the neuroscience viewpoint, and the computer science/engineering aspects.

A number of recent reviews [

113

130

172

] address the eld of BIDL from a high-level perspective, discussing

the mutual benets that exploration in biologically grounded mechanisms behind intelligence could bring both to the

neuroscience and computer science elds. The authors also suggest possible specic directions of exploration where

biological inspiration could play a crucial role.

A general background about biologically grounded neural system modeling can be found in the book from Gerstner

and Kistler [

], where a thorough introduction to SNN models is provided, as well as to Hebbian and Spike Time

Dependent Plasticity (STDP) models. A review more specically focused on reward-modulated Hebbian learning

approaches can be found in [

202

], while recent SNN developments and applications are surveyed in [

151

160

196

The discussions in [

148

] focus more specically on backprop alternatives for SNNs. Finally, various recent surveys

provide a thorough analysis of the eld of neuromorphic hardware [84, 174, 186, 190, 229].

Compared to previous surveys, we provide signicant contributions in the following directions:

•

We emphasize the connections between low-level learning mechanisms and high-level abstractions, as opposed

to works more focused on the high-level perspectives [73, 113, 130, 172];

•

We develop a comprehensive viewpoint involving bio-inspired techniques ranging from synaptic plasticity

models for spiking neurons, to supervised or reward-driven backprop alternatives, while other contributions

only focus on backprop extensions for SNNs [45, 148] or reward-driven approaches [202];

•

We highlight the connections between traditional and spike-based models, showing the interplay between the

two domains, and the potentials of SNN models for energy-ecient neuromorphic and biological computing,

compared to works that focus more specically on the low-level mechanisms of spike-based computation

[151, 160, 196].

Manuscript submitted to ACM

4 Lagani, et al.

3 BIOLOGICAL NEURONS AND SYNAPTIC PLASTICITY

This Section is devoted to introducing the fundamental aspects of neuron biology, and the mechanisms of synaptic

plasticity underlying the learning behavior of biological brains. From this it will be possible to draw relationships

between the computational learning mechanisms that we are going to discuss in the following, and the biological

substrate that supports such mechanisms.

In the following, we start with an introduction on neural cell biology in subsection 3.1, then we move to synaptic

plasticity models based on the Hebbian principle in subsection 3.2, highlighting the relationships between such models

and emerging unsupervised pattern discovery mechanisms (such as clustering and subspace learning).

3.1 Background on Neural Cells

Neural cells are characterized by a central body, the soma, which receives electric input stimuli through ramied

connections called dendrites, and emit signals through an output connection called axon [

124

188

213

]. The axon

is connected to other neurons’ dendrites by chemical couplings called synapses. The strength of synaptic couplings is

plastic, and its change determines the learning properties of neurons. The main aspects of plasticity are strengthening of

synaptic ecacy, a.k.a. Long Term Potentiation (LTP), or weakening, a.k.a. Long Term Depression (LTD) [

188

213

Upon receiving input stimuli, an electric potential is accumulated on the neuronal membrane (i.e. the membrane potential).

When the membrane potential exceeds a threshold, an output stimulus is triggered, in terms of a pulse-shaped (a spike)

electric signal propagated on the axon [61].

We can distinguish two main groups of neural cells: pyramidal and non-pyramidal [213].

Pyramidal cells represent the main computing unit in the brain. As the name suggest, they are characterized by a

pyramid-like shape, from which two types of dendritic trees branch out: apical and basal. The former extends from the

tip of the pyramid, and extend through cortical layers, while the latter extend mainly toward neighboring cells in the

same region. From the base of the pyramid, the axon of the neural cell originates. I then branches into a projection axon,

which can extend towards deeper cortical layers, and several axon collaterals, which can either be local, i.e. extending

for a short distance towards neighboring neurons, or they can extend for longer distances either in the same layer or

towards neighboring layers.

Non-pyramidal cells are instead a broad category comprising a variety of neural cell types [

194

], but all sharing

some common features such as the presence of a central body with a smaller size compared to pyramidal cells, the

local connectivity patterns (i.e. short-distance), and the mainly inhibitory role of such neurons. A dense dendritic tree

originates from the cell body, as well as an axon which tends to branch into multiple ramications. Due to the local

nature of non-pyramidal cell connectivity, which typically tend to target neighboring pyramidal cells, thus routing

information in the local neighborhood, these neurons are also often referred to as interneurons. Interneurons play an

important role in inhibitory interaction and shunting inhibition [

100

] between neurons. The mechanisms of inhibitory

lateral interaction, mediated by non-pyramidal cells, are essential to achieve decorrelation in neural activity. In the

following we will discuss how these mechanisms, coupled with appropriate synaptic plasticity models, allow neural

circuits to implement unsupervised algorithms for automatic pattern discovery, thus showing an interesting mapping

between biological circuits and learning mechanisms.

Manuscript submitted to ACM

Bio-Inspired Deep Learning 5

Fig. 2. Representation of a neuron model with weights

𝑤

, 𝑤

, taking inputs

𝑥

, 𝑥

, and computing output

𝑦 = 𝑓 (

𝑖=1

𝑥

𝑖

𝑤

𝑖

)

3.2 Biological Models of Synaptic Plasticity

In the following we will consider a neuron whose synaptic weights are described by a vector w. The neuron takes as

input a vector x, and produces an output

𝑦(

) = 𝑓 (

𝑇

)

(Fig. 2), where

𝑓

is the activation function (optionally, a

bias term can be implicitly modeled as a synapse connected to a constant input). In the following, we use boldface

fonts to denote vectors, and normal fonts to denote scalars. In neuroscientic terms, input values x are also termed

pre-synaptic activations, while the output

𝑦

is termed post-synaptic activation. In order to draw a relationship between

this abstract neuron model, and the biological neural cell structure described in the previous subsection, we can imagine

that the output value of the abstract neuron model represents the rate of output spikes generated by the neuron due to

the input stimuli, hence we can talk about a rate-coded neuron model [

]. Similarly, the input value on a given synapse

corresponds to the rate of input spikes delivered through that synapse.

In order to model synaptic plasticity, neuroscientists propose the Hebbian principle [

]: "re together, wire

together". According to this principle, the synaptic coupling between two neurons is reinforced when the two neurons

are simultaneously active [

]. Mathematically, this learning rule, in its simplest "vanilla" formulation, can be

expressed as:

Δ𝑤

𝑖

= 𝜂 𝑦 𝑥

𝑖

(1)

where

𝜂

is the learning rate, and the subscript

𝑖

refers to the i-th input/synapse. The eect of this learning rule

is essentially to consolidate correlated activations between neural inputs and outputs, by reinforcing the synaptic

couplings, so that, if a similar input will be observed again in the future, a similar response will likely be elicited from

the neuron.

The problem with Eq. 1 is that it is unstable, as repeated stimulation can induce the weights to grow unbounded.

This issue can be counteracted by iteratively normalizing the weight vector [

152

], or by adding a weight decay term

𝛾 (w, x) to the learning rule:

Δ𝑤

𝑖

= 𝜂 𝑦 𝑥

𝑖

− 𝛾 (w, x) (2)

In particular, with an appropriate choice for this term, i.e.

𝛾 (

) = 𝜂 𝑦(

) 𝑤

𝑖

, we obtain a learning rule that has

been widely applied to the context of competitive learning [67, 96, 178]:

Δ𝑤

𝑖

= 𝜂 𝑦 𝑥

𝑖

− 𝜂 𝑦 𝑤

𝑖

= 𝜂 𝑦 (𝑥

𝑖

− 𝑤

𝑖

) (3)

It can be shown that learning rule in Eq. 1 induces the weight vector of a neuron to align towards the principal

component of the data distribution it is trained on [

152

], while Eq. 3 pushes the weight vector towards the centroid of

the cluster formed by the data points [

]. This shows surprising connections between biological models of synaptic

plasticity and computer science aspects related to data engineering and unsupervised pattern discovery mechanisms,

specically Principal Component Analysis (PCA) and clustering.

Manuscript submitted to ACM

剩余30页未读，继续阅读

评论收藏

内容反馈

学术菜鸟小晨

粉丝: 1w+
资源: 4953

Spiking Neural Networks and Bio-Inspired Supervised Deep Learnin

Time-Space, Spiking Neural Networks and Brain-Inspired Artificial Intelligence

Spiking neural networks.rar_Neural networks_SNN_Spiking-Neural_s

Deep Spiking Neural Network for Video-Based Disguise Face Recognition

Spiking Neural Networks: The Machine Learning Approach

藏经阁-Spiking Neural Networks, the N.pdf

Patio-Temporal Backpropagation for Training High-Performance Spiking Neural Networks.pdf

Training-spiking-neural-networks-for-reinforcement-learning:源代码：https

NeMo_脉冲神经网络工具_spiking neural network

多层卷积脉冲神经网络.pdf

Hebbian Learning in Spiking Neural Networks

Spiking neural networks

关于GPU对spiking neural network的提速，2012年的，值得一看！

在 PyTorch 中使用 spiking neural networks (SNN) 进行深度学习_spikingneuron

用于分层强化学习的随机神经网络 SNN_ spiking neural networks_spikingneuron

Supervised Learning Algorithm for Spiking Neurons Based on Nonlinear Inner Products of Spike Trains

PyTorch-Spiking-YOLOv3:Spiking-YOLOv3的PyTorch实现。 根据YOLOv3的两个常见PyTorch实现（ultralyticsyolov3和eriklindernorenPyTorch-YOLOv3）提供了两个分支，目前支持Spiking-YOLOv3-Tiny

使用 Spiking 神经网络的 One-Shot 对象学习spikingneuron_LIF_neuron_LIF神经元

YOLOv8-deepsort 实现智能车辆目标检测+车辆跟踪+车辆计数

YOLOv8网络结构图，自制visio文件，yolov8.vsds，需要的自取，在原有的基础上直接改就行了

yolov8(2023年8月版本),已经下好yolov8s.pt和yolov8n.pt

Transformer模型实现长期预测并可视化结果（附代码+数据集+原理介绍）

社交平台上经济类话题的文章热度信息，数据是真实的，但不是真实日期

行人跌倒数据集（VOC格式）

Unet眼底血管图像分割数据集+代码+模型+系统界面+教学视频.zip

全新的SOTA模型YOLOv9

YOLOV5 + 双目相机实现三维测距（新版本）

YOLOV5口罩检测数据集+代码+模型 2000张标注好的数据+教学视频.zip

pycharm连接autodl服务器（yolov8训练自己的数据集）

最新资源

PyTorch-Spiking-YOLOv3:Spiking-YOLOv3的PyTorch实现。根据YOLOv3的两个常见PyTorch实现（ultralyticsyolov3和eriklindernorenPyTorch-YOLOv3）提供了两个分支，目前支持Spiking-YOLOv3-Tiny