# Transformer Models for MATLAB
[![CircleCI](https://img.shields.io/circleci/build/github/matlab-deep-learning/transformer-models?label=tests)](https://app.circleci.com/pipelines/github/matlab-deep-learning/transformer-models)
[![Open in MATLAB Online](https://www.mathworks.com/images/responsive/global/open-in-matlab-online.svg)](https://matlab.mathworks.com/open/github/v1?repo=matlab-deep-learning/transformer-models)
This repository implements deep learning transformer models in MATLAB.
## Translations
* [日本語](./README_JP.md)
## Requirements
### BERT and FinBERT
- MATLAB R2021a or later
- Deep Learning Toolbox
- Text Analytics Toolbox
### GPT-2
- MATLAB R2020a or later
- Deep Learning Toolbox
## Getting Started
Download or [clone](https://www.mathworks.com/help/matlab/matlab_prog/use-source-control-with-projects.html#mw_4cc18625-9e78-4586-9cc4-66e191ae1c2c) this repository to your machine and open it in MATLAB.
## Functions
### bert
`mdl = bert` loads a pretrained BERT transformer model and if necessary, downloads the model weights. The output `mdl` is structure with fields `Tokenizer` and `Parameters` that contain the BERT tokenizer and the model parameters, respectively.
`mdl = bert("Model",modelName)` specifies which BERT model variant to use:
- `"base"` (default) - A 12 layer model with hidden size 768.
- `"multilingual-cased"` - A 12 layer model with hidden size 768. The tokenizer is case-sensitive. This model was trained on multi-lingual data.
- `"medium"` - An 8 layer model with hidden size 512.
- `"small"` - A 4 layer model with hidden size 512.
- `"mini"` - A 4 layer model with hidden size 256.
- `"tiny"` - A 2 layer model with hidden size 128.
- `"japanese-base"` - A 12 layer model with hidden size 768, pretrained on texts in the Japanese language.
- `"japanese-base-wwm"` - A 12 layer model with hidden size 768, pretrained on texts in the Japanese language. Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective.
### bert.model
`Z = bert.model(X,parameters)` performs inference with a BERT model on the input `1`-by-`numInputTokens`-by-`numObservations` array of encoded tokens with the specified parameters. The output `Z` is an array of size (`NumHeads*HeadSize`)-by-`numInputTokens`-by-`numObservations`. The element `Z(:,i,j)` corresponds to the BERT embedding of input token `X(1,i,j)`.
`Z = bert.model(X,parameters,Name,Value)` specifies additional options using one or more name-value pairs:
- `"PaddingCode"` - Positive integer corresponding to the padding token. The default is `1`.
- `"InputMask"` - Mask indicating which elements to include for computation, specified as a logical array the same size as `X` or as an empty array. The mask must be false at indices positions corresponds to padding, and true elsewhere. If the mask is `[]`, then the function determines padding according to the `PaddingCode` name-value pair. The default is `[]`.
- `"DropoutProb"` - Probability of dropout for the output activation. The default is `0`.
- `"AttentionDropoutProb"` - Probability of dropout used in the attention layer. The default is `0`.
- `"Outputs"` - Indices of the layers to return outputs from, specified as a vector of positive integers, or `"last"`. If `"Outputs"` is `"last"`, then the function returns outputs from the final encoder layer only. The default is `"last"`.
- `"SeparatorCode"` - Separator token specified as a positive integer. The default is `103`.
### finbert
`mdl = finbert` loads a pretrained BERT transformer model for sentiment analysis of financial text. The output `mdl` is structure with fields `Tokenizer` and `Parameters` that contain the BERT tokenizer and the model parameters, respectively.
`mdl = finbert("Model",modelName)` specifies which FinBERT model variant to use:
- `"sentiment-model"` (default) - The fine-tuned sentiment classifier model.
- `"language-model"` - The FinBERT pretrained language model, which uses a BERT-Base architecture.
### finbert.sentimentModel
`sentiment = finbert.sentimentModel(X,parameters)` classifies the sentiment of the input `1`-by-`numInputTokens`-by-`numObservations` array of encoded tokens with the specified parameters. The output sentiment is a categorical array with categories `"positive"`, `"neutral"`, or `"negative"`.
`[sentiment, scores] = finbert.sentimentModel(X,parameters)` also returns the corresponding sentiment scores in the range `[-1 1]`.
### gpt2
`mdl = gpt2` loads a pretrained GPT-2 transformer model and if necessary, downloads the model weights.
### generateSummary
`summary = generateSummary(mdl,text)` generates a summary of the string or `char` array `text` using the transformer model `mdl`. The output summary is a char array.
`summary = generateSummary(mdl,text,Name,Value)` specifies additional options using one or more name-value pairs.
* `"MaxSummaryLength"` - The maximum number of tokens in the generated summary. The default is 50.
* `"TopK"` - The number of tokens to sample from when generating the summary. The default is 2.
* `"Temperature"` - Temperature applied to the GPT-2 output probability distribution. The default is 1.
* `"StopCharacter"` - Character to indicate that the summary is complete. The default is `"."`.
## Example: Classify Text Data Using BERT
The simplest use of a pretrained BERT model is to use it as a feature extractor. In particular, you can use the BERT model to convert documents to feature vectors which you can then use as inputs to train a deep learning classification network.
The example [`ClassifyTextDataUsingBERT.m`](./ClassifyTextDataUsingBERT.m) shows how to use a pretrained BERT model to classify failure events given a data set of factory reports. This example requires the `factoryReports.csv` data set from the Text Analytics Toolbox example [Prepare Text Data for Analysis](https://www.mathworks.com/help/textanalytics/ug/prepare-text-data-for-analysis.html).
## Example: Fine-Tune Pretrained BERT Model
To get the most out of a pretrained BERT model, you can retrain and fine tune the BERT parameters weights for your task.
The example [`FineTuneBERT.m`](./FineTuneBERT.m) shows how to fine-tune a pretrained BERT model to classify failure events given a data set of factory reports. This example requires the `factoryReports.csv` data set from the Text Analytics Toolbox example [Prepare Text Data for Analysis](https://www.mathworks.com/help/textanalytics/ug/prepare-text-data-for-analysis.html).
The example [`FineTuneBERTJapanese.m`](./FineTuneBERTJapanese.m) shows the same workflow using a pretrained Japanese-BERT model. This example requires the `factoryReportsJP.csv` data set from the Text Analytics Toolbox example [Analyze Japanese Text Data](https://www.mathworks.com/help/textanalytics/ug/analyze-japanese-text.html), available in R2023a or later.
## Example: Analyze Sentiment with FinBERT
FinBERT is a sentiment analysis model trained on financial text data and fine-tuned for sentiment analysis.
The example [`SentimentAnalysisWithFinBERT.m`](./SentimentAnalysisWithFinBERT.m) shows how to classify the sentiment of financial news reports using a pretrained FinBERT model.
## Example: Predict Masked Tokens Using BERT and FinBERT
BERT models are trained to perform various tasks. One of the tasks is known as masked language modeling which is the task of predicting tokens in text that have been replaced by a mask value.
The example [`PredictMaskedTokensUsingBERT.m`](./PredictMaskedTokensUsingBERT.m) shows how to predict masked tokens and calculate the token probabilities using a pretrained BERT model.
The example [`PredictMaskedTokensUsingFinBERT.m`](./PredictMaskedTokensUsingFinBERT.m) shows how to predict masked tokens for financial text using and calculate the token probabilities using a pretrained FinBERT model.
## Example: Summarize Text Using GPT-2
Transformer networks such as GPT-2 can be used to summarize a piece of text. The trained GPT-2 trans
没有合适的资源?快使用搜索试试~ 我知道了~
毕业设计&课设-MATLAB中的深度学习变压器模型.zip
共98个文件
m:92个
md:3个
yml:1个
0 下载量 54 浏览量
2024-01-08
22:40:50
上传
评论
收藏 104KB ZIP 举报
温馨提示
matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随时与博主沟通,第一时间进行解答! matlab算法,工具源码,适合毕业设计、课程设计作业,所有源码均经过严格测试,可以直接运行,可以放心下载使用。有任何使用问题欢迎随
资源推荐
资源详情
资源评论
收起资源包目录
毕业设计&课设-MATLAB中的深度学习变压器模型.zip (98个子文件)
matlab_code
SECURITY.md 387B
predictMaskedToken.m 2KB
+transformer
+layer
dropout.m 308B
attention.m 5KB
multiheadAttention.m 3KB
multiLayerPerceptron.m 1KB
gelu.m 461B
normalization.m 785B
maskAttentionWeights.m 2KB
convolution1d.m 538B
FineTuneBERT.m 10KB
finbert.m 1KB
.circleci
config.yml 489B
ClassifyTextDataUsingBERT.m 8KB
PredictMaskedTokensUsingFinBERT.m 3KB
gpt2.m 378B
FineTuneBERTJapanese.m 11KB
generateSummary.m 4KB
+bert
languageModel.m 805B
+tokenizer
BERTTokenizer.m 11KB
+internal
WhitespaceTokenizer.m 589B
Tokenizer.m 138B
TokenizedDocumentTokenizer.m 1010B
FullTokenizer.m 6KB
BasicTokenizer.m 6KB
WordPieceTokenizer.m 4KB
+layer
block.m 3KB
languageModelHead.m 1012B
pooler.m 627B
embedding.m 1KB
classifierHead.m 537B
classifier.m 583B
load.m 600B
model.m 5KB
+internal
createParameterStruct.m 4KB
predictMaskedToken.m 325B
inferTypeID.m 588B
encodeWithMaskToken.m 998B
getSupportFilePath.m 701B
convertModelNameToDirectories.m 1KB
PredictMaskedTokensUsingBERT.m 3KB
+gpt2
+tokenizer
GPT2Tokenizer.m 11KB
+layer
block.m 4KB
load.m 1KB
model.m 4KB
+internal
getSupportFilePath.m 2KB
download.m 2KB
test
tfinbert.m 872B
sampling
tsampleFromCategorical.m 1KB
ttopKLogits.m 2KB
tools
DownloadGPT2Fixture.m 2KB
DownloadJPBERTFixture.m 2KB
DownloadFinBERTFixture.m 2KB
DownloadBERTFixture.m 2KB
getRepoRoot.m 307B
transformer
layer
tmaskAttentionWeights.m 1KB
tmultiLayerPerceptron.m 2KB
tgelu.m 2KB
tconvolution1d.m 1KB
tattention.m 13KB
tmultiheadAttention.m 10KB
tdropout.m 2KB
tnormalization.m 2KB
tpredictMaskedToken.m 2KB
gpt2
tload.m 2KB
tokenizer
tGPT2Tokenizer.m 7KB
tmodel.m 2KB
tdownload.m 1KB
layer
tblock.m 4KB
finbert
tsentimentModel.m 1KB
tlanguageModel.m 2KB
tbert.m 3KB
bert
tokenizer
internal
tTokenizedDocumentTokenizer.m 1KB
tBasicTokenizer.m 3KB
tWhitespaceTokenizer.m 499B
tWordPieceTokenizer.m 3KB
tFullTokenizer.m 2KB
tBERTTokenizer.m 8KB
tBERTTokenizerForJP.m 2KB
tmodel.m 12KB
internal
tinferTypeID.m 2KB
tlanguageModel.m 4KB
.gitignore 15B
README_JP.md 10KB
SentimentAnalysisWithFinBERT.m 2KB
bert.m 2KB
SummarizeTextUsingTransformersExample.m 1KB
+sampling
topKLogits.m 1KB
sampleFromCategorical.m 661B
license.txt 1KB
README.md 8KB
+finbert
languageModel.m 603B
sentimentModel.m 931B
load.m 584B
model.m 4KB
+internal
getSupportFilePath.m 704B
convertModelNameToDirectories.m 289B
truncateSequences.m 2KB
共 98 条
- 1
资源评论
白话机器学习
- 粉丝: 9291
- 资源: 7681
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功