Mixture-of-ExpertswithExpertChoiceRouting.pdf资源-CSDN文库

需积分: 5 71 浏览量 2024-06-15 07:41:19 上传评论收藏 592KB PDF 举报

### 混合专家模型与专家选择路由技术详解 #### 一、引言与背景在深度学习领域，特别是自然语言处理（NLP）任务中，混合专家模型（Mixture-of-Experts, MoE）因其能够有效利用大量参数而受到广泛关注。通过允许参数数量大幅增加的同时保持每个样本或令牌所需的计算量不变，MoE模型能够在不显著增加计算资源消耗的情况下提高模型性能。然而，传统的专家路由策略可能导致某些专家训练不足，从而影响模型的整体表现。 #### 二、传统MoE模型的问题与挑战传统的MoE模型通常采用固定数量的专家对每个令牌进行分配，通常使用top-k函数来选择最相关的k个专家。这种方法忽略了不同令牌之间的重要性差异，导致某些专家可能被过度或不足地利用，即过度专业化或训练不足。 1. **专家训练不均衡**：由于令牌根据其需求选择专家，这可能导致某些专家接收到的数据量远少于其他专家，进而影响这些专家的学习效果。 2. **过度或不足专业化**：固定的专家分配机制可能会导致某些专家处理过于狭窄或过于广泛的输入数据，影响模型的泛化能力。 #### 三、专家选择路由方法为了解决上述问题，研究人员提出了异构混合专家模型（Heterogeneous Mixture-of-Experts, HMoE），该模型采用了专家选择令牌的方法，即让专家选择它们想要处理的令牌。这种方法确保了每个令牌可以被路由到不同数量的专家，同时保证每个专家具有固定大小的数据集。 1. **专家主动选择令牌**：每个专家可以选择处理自己认为最重要的前k个令牌，这样可以确保每个专家都能够专注于最具代表性和重要的输入数据。 2. **灵活的令牌分配**：每个令牌可以根据其特定需求被分配给不同数量的专家，从而更好地满足模型训练的需求。 #### 四、实验结果与分析研究人员通过对Switch Transformer top-1和GShard top-2两种不同类型的MoE模型进行预训练，并使用相同的计算资源进行了对比实验。实验结果表明，采用专家选择令牌的方法可以显著提高模型的训练收敛速度，比传统的top-k方法提高了超过2倍。此外，在GLUE和SuperGLUE这两个广泛使用的自然语言处理基准测试中，采用专家选择令牌的方法不仅在相同的计算成本下取得了更高的性能，而且在更小的激活成本下也超过了T5密集模型在7项任务中的表现。 #### 五、结论与展望通过采用专家选择令牌的方法，不仅可以解决传统MoE模型中存在的专家训练不均衡问题，还能显著提高模型的训练效率和性能。这一改进对于未来大规模自然语言处理模型的设计和优化具有重要意义。随着研究的深入和技术的进步，我们有望看到更多创新的路由策略应用于实际场景，进一步推动自然语言处理领域的技术发展。

资源推荐

资源详情

资源评论

Mixture-of-Experts with Expert Choice Routing

Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng

Chen, Quoc Le, and James Laudon

Google, Mountain View, CA, USA

{yanqiz, taole, hanxiaol, dunan, huangyp, vzhao, adai, zhifengc, qvl,

jlaudon}@google.com

Abstract

Sparsely-activated Mixture-of-experts (MoE) models allow the number of param-

eters to greatly increase while keeping the amount of computation for a given

token or a given sample unchanged. However, a poor expert routing strategy can

cause certain experts to be under-trained, leading to an expert being under or

over-specialized. Prior work allocates a ﬁxed number of experts to each token

using a top-

function regardless of the relative importance of different tokens. To

address this, we propose a heterogeneous mixture-of-experts employing an expert

choice method. Instead of letting tokens select the top-

experts, we have experts

selecting the top-

tokens. As a result, each token can be routed to a variable

number of experts and each expert can have a ﬁxed bucket size. We systematically

study pre-training speedups using the same computational resources of the Switch

Transformer top-1 and GShard top-2 gating of prior work and ﬁnd that our method

improves training convergence time by more than 2

. For the same computational

cost, our method demonstrates higher performance in ﬁne-tuning 11 selected tasks

in the GLUE and SuperGLUE benchmarks. For a smaller activation cost, our

method outperforms the T5 dense model in 7 out of the 11 tasks.

1 Introduction

Scaling up model capacity, dataset size, and training time has demonstrated huge success in enhancing

the performance of computer vision architectures [

] as well as neural language models [

]. The ﬁnal model quality has been found to have a power-law relationship with the amount

of data, model size, and compute time [

]. However, training efﬁciency, which is deﬁned as

the total amount of computation used to achieve superior model quality than the state of the art

system [21], should receive greater attention as we increase our efforts towards green AI [29].

Sparsely gated mixture-of-experts [

] (MoE) provides an effective way to scale model capacity

given a ﬁxed computational cost, and has recently played an important role in increasing the training

efﬁciency of large-scale language models [

]. MoE operate by adopting a number of experts,

each as a sub-network, and by activating only one or a few experts for each input token. A gating

network must be chosen and optimized in order to route each token to the most suited expert(s). For

example, recent work has implemented sparse routing via

-means clustering [

], linear assignment

to maximize token-expert afﬁnities [

], or hashing [

]. Many of the prior work use a routing

strategy concerning the token choice, where each token selects the best one or two experts.

We argue that the independent token choice of prior work often leads to an imbalanced load of experts,

which causes training inefﬁciency and sub-optimal training of the model. In order to mitigate this

36th Conference on Neural Information Processing Systems (NeurIPS 2022).

arXiv:2202.09368v2 [cs.LG] 14 Oct 2022

FFN 1 FFN 2 FFN 3 FFN 4 FFN 1 FFN 2 FFN 3 FFN 4

Router Router

Token 1

Token 2

We Like

p = 0.8

p = 0.65

Router

Token 1

Token 2

We Like To Play Soccer In The Field

We Like To

Play

Token 3

Token 4

Token 5

Token 6

Token 7

Token 8

We Like

Soccer

Field

FFN 1 FFN 2

Top-k Top-k

Figure 1: High-level Comparison Between Conventional MoE and expert choice MoE.

issue, previous sparsely gated networks introduce additional auxiliary losses as regularization to

prevent too many tokens being routed to a single expert, but the effectiveness is still limited. Recent

approaches [

] explore alternative strategies for routing, but they focus on pre-training only

and do not demonstrate performance gain on downstream tasks. Moreover, none of the previous

methods consider allocating a variable number of experts to each token based on importance, which

can be beneﬁcial.

We propose a very simple yet effective routing method we are calling expert choice. Unlike conven-

tional MoE where tokens select one or two top-scoring experts, our method lets each expert pick the

top-

tokens. Our method guarantees perfect load balancing, allows a variable number of experts

for each token, and achieves substantial gains in training efﬁciency and downstream performance as

demonstrated in our experiments. Our major contributions include:

•

We identify common pitfalls in conventional MoE such as load imbalance as described

in Section 3.1. We then propose a heterogeneous, expert choice method to provide a ﬂuid

allocation of model parameters based on a learnt token-to-expert importance. This method

intrinsically guarantees load balance without imposing an auxiliary loss.

•

We show our method provides over 2

faster training convergence in a 8B/64E (8 billion

activated parameters, 64 experts) model, compared to the top-1 and top-2 gating counterparts

in Switch Transformer [10] and GShard [21].

•

We show our method demonstrates strong scaling when increasing the number of experts

from 16 to 128, evaluated in training perplexity.

•

We show our method demonstrates strong performance on downstream tasks selected from

GLUE and SuperGLUE at all the evaluated scales. More speciﬁcally, our 8B/64E model

outperforms a T5 11B dense model in 7 out of 11 tasks evaluated.

2 Related Work

Scaling:

Various approaches have been proposed to scale up neural network capacity to improve

performance. Recent works have successfully scaled models to billions of parameters via various

forms of model parallelism [

]. Model parallelism [

] splits weights and tensors

across multiple cores while pipeline parallelism [

] splits different layers across devices with

micro-batches pipelined to the different layers. To enable continued scaling of neural networks,

improving model training and serving efﬁciency has become a critical research area.

Conditional Computation:

Computation decisions can be made dynamically based on the input [

]. Conditional computation has been proposed as a way to increase the capacity of a deep

neural network without increasing the amount of computation, by activating certain parameters and

computation on demand, on a per-example or per-token basis [

]. Conditional convolution layers [

]

with task-speciﬁc gating has been used to combat catastrophic forgetting when a sequence of learning

problems are optimized. The gating decisions may be binary or sparse and continuous, stochastic or

deterministic.

Mixture of Experts:

Sparsely-gated MoE [

] is the ﬁrst model to demonstrate massive improve-

ments in model capacity, training time, or model quality with gating. Switch Transformer [

]

simpliﬁes the gating by selecting only the top expert per token using a softmax over the hidden state

and demonstrates better scaling than previous work. All the prior work requires an auxiliary loss to

explicitly encourage balancing. This loss term has to be carefully weighted to not overwhelm the

primary loss. However, auxiliary loss does not guarantee balancing and a hard capacity factor has to

be imposed. As a result, many tokens can still be unprocessed by the MoE layer. Hard MoE [

] with

a single decoding layer can be efﬁciently trained to good effect on large scale hashtag prediction tasks.

Base Layers [

] formulate a linear assignment that maximizes token-expert afﬁnities while ensuring

each expert receives an equal number of tokens. Hash layers [

] devise hashing techniques on

input tokens. However, the evaluations are limited to pre-training perplexity. THOR [

] randomly

activates experts during training and inference and is trained with a consistency regularization loss.

THOR has demonstrated strong performance on translation tasks. Different from these prior works,

our method is a learnt method that enables heterogeneous MoE and effectively improves downstream

ﬁne-tuning performance.

3 Method

We ﬁrst identify a few pitfalls in the routing method of conventional mixture-of-experts (MoE) models

and then present our method using expert choice to tackle these problems.

3.1 Pitfalls of Token-Choice Routing

MoE can be computationally advantageous compared to a dense model, a routing strategy must be

used to assign each token to the most-suited experts. Conventional MoE models employ token-choice

routing which independently selects the top-

experts for each token [

]. We argue that this

strategy has a few pitfalls that lead to sub-optimal training.

Load Imbalance:

Token-choice routing often lead to poor load balancing across experts. That is,

some experts may be trained with most tokens, leaving the remaining experts under-utilized. Experts

can be under specialized because a lot of model capacity in the under-utilized experts are wasted.

On the other side, some tokens will not be processed, since over-utilized experts can only take a

maximum number of tokens at each step in order to avoid running out of memory. Load imbalance can

also hurt step latency, thus inference time, as the step latency can be determined by the most loaded

expert. Previous methods add an auxiliary loss on load balancing to mitigate the issue. However, this

auxiliary loss does not guarantee a balanced load, especially during the important early stages of

training. Indeed,

we empirically observe that the over-capacity ratio can reach 20%–40% for

some experts in token choice routing

, indicating that a signiﬁcant portion of the tokens routed to

these experts will be dropped.

Under Specialization:

Each MoE layer uses a gating network to learn token-to-expert afﬁnity.

Ideally, the learnt gating network should produce the afﬁnity such that similar or relevant tokens are

routed to the same expert. A sub-optimal strategy can produce redundant experts and/or experts that

are not sufﬁciently specialized. Under specialization may result by imposing an large auxiliary loss

which favors more load balanced but less effective routing. Finding the right balance on the auxiliary

loss to promote both load balancing and specialization is challenging for token-choice routing.

Same Compute for Every Token:

Finally, in a token-choice strategy each token receives exactly

experts and therefore occupies the same amount of compute. We hypothesize that this is not

necessary nor desired. Instead, a MoE model should ﬂexibly allocate its compute resource based

on the complexity of the input. Motivated by the aforementioned observations, we next describe a

simple yet effective method which produces load balanced assignments based on expert choice.

3.2 Heterogeneous MoE via Expert Choice

Different from conventional routing, an expert choice method independently selects top-

tokens

for each expert, where

is a ﬁxed expert capacity (i.e. the number of tokens each expert can take).

Despite its simplicity, expert choice achieves perfect load balancing by design. It also enables more

ﬂexible allocation of model compute since tokens can be received by a variable number of experts.

剩余13页未读，继续阅读

评论收藏

内容反馈

（initial）

粉丝: 294
资源: 34

Mixture-of-Experts with Expert Choice Routing.pdf

最新资源

Mixture-of-Experts with Expert Choice Routing.pdf

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

mixture-of-experts:混合Keras专家层

mixture-of-experts:Noam Shazeer等人的“稀疏门控专家混合物层”的PyTorch重新实现。 https

Kernel-Based Mixture of Experts Models For Linear Regression:Kernel-Based Mixture of Experts Models For Linear Regression-matlab开发

Python库 | mixture_of_experts-0.1.0-py3-none-any.whl

基于混合高斯模型的运动目标检测问题-mixture_of_gaussians.rar

Comparison of the finite mixture of ARMA-GARCH NN SVM financial returns..pdf

混合高斯模型-mixture_of_gaussians.m

Gaussian-Mixture-Models-master.zip

Stauffer1999A_Adaptive background mixture models for real-time tracking.pdf

Gaussian-mixture-model-algorithm.rar_Gaussian Mixture_mixture_混合

t-Student-Mixture-Models:论文执行

IC.zip_ICA MATLAB_mixture- code._the code

ST-MOE DESIGNING STABLE AND TRANSFERABLE SPARSE EXPERT MODEL.pdf

2023年美赛特等奖论文-C-2307946-解密.pdf

Gaussian-Mixture-Model-master_matlab_

Piecewise-Cox-Mixture-Cure-Model-with-IPTW:统一的两步程序解决不平衡处理，严格的检查和聚类

K均值聚类，多维度

expert choice教程

expert choice

Expert Choice

Adaptive-background-mixture-models-for-real-time-_For Real_adapt

击败GPT3，刷新50个SOTA！谷歌全面统一NLP范式.pdf

PyPI 官网下载 | mixture-0.1.0-py3-none-any.whl

em算法matlab代码-Expectation-Maximization-with-Gaussian-Mixture-Model:高斯混合模

Text Mining: Classification, Clustering, and Applications

Hidden Markov Model (HMM) matlab程序

最新资源