Google开源大模型gemma技术报告
需积分: 0 29 浏览量
2024-03-10
10:24:26
上传
评论
收藏 445KB PDF 举报
2024-02-21
Gemma: Open Models Based on Gemini
Research and Technology
Gemma Team, Google DeepMind
1
1
See Contributions and Acknowledgments section for full author list. Please send correspondence to [email protected].
This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research
and technology used to create Gemini models. Gemma models demonstrate strong performance across
academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models
(2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma
outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive
evaluations of safety and responsibility aspects of the models, alongside a detailed description of model
development. We believe the responsible release of LLMs is critical for improving the safety of frontier
models, and for enabling the next wave of LLM innovations.
Introduction
We present Gemma, a family of open models
based on Google’s Gemini models (Gemini Team,
2023).
We trained Gemma models on up to 6T to-
kens of text, using similar architectures, data,
and training recipes as the Gemini model family.
Like Gemini, these models achieve strong gener-
alist capabilities in text domains, alongside state-
of-the-art understanding and reasoning skills at
scale. With this work, we release both pre-trained
and fine-tuned checkpoints, as well as an open-
source codebase for inference and serving.
Gemma comes in two sizes: a 7 billion param-
eter model for efficient deployment and develop-
ment on GPU and TPU, and a 2 billion param-
eter model for CPU and on-device applications.
Each size is designed to address different compu-
tational constraints, applications, and developer
requirements. At each scale, we release raw, pre-
trained checkpoints, as well as checkpoints fine-
tuned for dialogue, instruction-following, help-
fulness, and safety. We thoroughly evaluate the
shortcomings of our models on a suite of quantita-
tive and qualitative benchmarks. We believe the
release of both pretrained and fine-tuned check-
points will enable thorough research and inves-
tigation into the impact of current instruction-
tuning regimes, as well as the development of
increasingly safe and responsible model develop-
ment methodologies.
Gemma advances state-of-the-art performance
relative to comparable-scale (and some larger),
open models (Almazrouei et al., 2023; Jiang
et al., 2023; Touvron et al., 2023a,b) across a
wide range of domains including both automated
benchmarks and human evaluation. Example do-
mains include question answering (Clark et al.,
2019; Kwiatkowski et al., 2019), commonsense
reasoning (Sakaguchi et al., 2019; Suzgun et al.,
2022), mathematics and science (Cobbe et al.,
2021; Hendrycks et al., 2020), and coding (Austin
et al., 2021; Chen et al., 2021). See complete de-
tails in the Evaluation section.
Like Gemini, Gemma builds on recent work
on sequence models (Sutskever et al., 2014) and
transformers (Vaswani et al., 2017), deep learn-
ing methods based on neural networks (LeCun
et al., 2015), and techniques for large-scale train-
ing on distributed systems (Barham et al., 2022;
Dean et al., 2012; Roberts et al., 2023). Gemma
also builds on Google’s long history of open mod-
els and ecosystems, including Word2Vec (Mikolov
et al., 2013), the Transformer (Vaswani et al.,
2017), BERT (Devlin et al., 2018), and T5 (Raffel
et al., 2019) and T5X (Roberts et al., 2022).
We believe the responsible release of LLMs is
critical for improving the safety of frontier models,
for ensuring equitable access to this breakthrough
technology, for enabling rigorous evaluation and
analysis of current techniques, and for enabling
© 2024 Google DeepMind. All rights reserved
剩余15页未读，继续阅读
资源评论
