<p align="center">
<br>
<img src="https://raw.githubusercontent.com/voidful/TFkit/master/docs/img/tfkit.png" width="300"/>
<br>
</p>
<br/>
<p align="center">
<a href="https://pypi.org/project/tfkit/">
<img alt="PyPI" src="https://img.shields.io/pypi/v/tfkit">
</a>
<a href="https://github.com/voidful/tfkit">
<img alt="Download" src="https://img.shields.io/pypi/dm/tfkit">
</a>
<a href="https://github.com/voidful/tfkit">
<img alt="Build" src="https://img.shields.io/github/workflow/status/voidful/tfkit/Python package">
</a>
<a href="https://github.com/voidful/tfkit">
<img alt="Last Commit" src="https://img.shields.io/github/last-commit/voidful/tfkit">
</a>
<a href="https://www.codefactor.io/repository/github/voidful/tfkit/overview/master">
<img src="https://www.codefactor.io/repository/github/voidful/tfkit/badge/master" alt="CodeFactor" />
</a>
<a href="https://github.com/voidful/tfkit">
<img src="https://visitor-badge.glitch.me/badge?page_id=voidful.tfkit" alt="Visitor" />
</a>
<a href="https://codecov.io/gh/voidful/TFkit">
<img src="https://codecov.io/gh/voidful/TFkit/branch/master/graph/badge.svg" />
</a>
</p>
TFKit lets everyone make use of transformer architecture on many tasks and models in small change of config.
At the same time, it can do multi-task multi-model learning, and can introduce its own data sets and tasks through simple modifications.
## Feature
- One-click replacement of different pre-trained models
- Support multi-model and multi-task
- Classifier with multiple labels and multiple classifications
- Unify input formats for different tasks
- Separation of data reading and model architecture
- Support various loss function and indicators
## Supplement
- [Model list](https://huggingface.co/models): Support Bert/GPT/GPT2/XLM/XLNet/RoBERTa/CTRL/ALBert/...
- [NLPrep](https://github.com/voidful/NLPrep): download and preprocessing data in one line
- [nlp2go](https://github.com/voidful/nlp2go): create demo api as quickly as possible.
# Documentation
Learn more from the [docs](https://voidful.github.io/TFkit/).
## Quick Start
### Installing via pip
```bash
pip install tfkit
```
### Running TFKit to train a ner model
install nlprep and nlp2go
```bash
pip install nlprep nlp2go -U
```
download dataset using nlprep
```bash
nlprep --dataset tag_clner --outdir ./clner_row --util s2t
```
train model with albert
```bash
tfkit-train --batch 20 \
--epoch 5 \
--lr 5e-5 \
--train ./clner_row/clner-train.csv \
--test ./clner_row/clner-test.csv \
--maxlen 512 \
--model tagRow \
--savedir ./albert_ner \
--config voidful/albert_chinese_small
```
eval model
```bash
tfkit-eval --model ./albert_ner/3.pt --valid ./clner_row/validation.csv --metric clas
```
result
```text
Task : default report
TASK: default 0
precision recall f1-score support
B_Abstract 0.00 0.00 0.00 1
B_Location 1.00 1.00 1.00 1
B_Metric 1.00 1.00 1.00 1
B_Organization 0.00 0.00 0.00 1
B_Person 1.00 1.00 1.00 1
B_Physical 0.00 0.00 0.00 1
B_Thing 1.00 1.00 1.00 1
B_Time 1.00 1.00 1.00 1
I_Abstract 1.00 1.00 1.00 1
I_Location 1.00 1.00 1.00 1
I_Metric 1.00 1.00 1.00 1
I_Organization 0.00 0.00 0.00 1
I_Person 1.00 1.00 1.00 1
I_Physical 0.00 0.00 0.00 1
I_Thing 1.00 1.00 1.00 1
I_Time 1.00 1.00 1.00 1
O 1.00 1.00 1.00 1
micro avg 1.00 0.71 0.83 17
macro avg 0.71 0.71 0.71 17
weighted avg 0.71 0.71 0.71 17
samples avg 1.00 0.71 0.83 17
```
host prediction service
```bash
nlp2go --model ./albert_ner/3.pt --api_path ner
```
**You can also try tfkit in Google Colab: [![Google Colab](https://colab.research.google.com/assets/colab-badge.svg "tfkit")](https://colab.research.google.com/drive/1hqaTKxd3VtX2XkvjiO0FMtY-rTZX30MJ?usp=sharing)**
## Overview
### Train
```
$ tfkit-train
Run training
arguments:
--train TRAIN [TRAIN ...] train dataset path
--test TEST [TEST ...] test dataset path
--config CONFIG distilbert-base-multilingual-cased/bert-base-multilingual-cased/voidful/albert_chinese_small
--model {once,twice,onebyone,clas,tagRow,tagCol,qa,onebyone-neg,onebyone-pos,onebyone-both} [{once,twice,onebyone,clas,tagRow,tagCol,qa,onebyone-neg,onebyone-pos,onebyone-both} ...]
model task
--savedir SAVEDIR model saving dir, default /checkpoints
optional arguments:
-h, --help show this help message and exit
--batch BATCH batch size, default 20
--lr LR [LR ...] learning rate, default 5e-5
--epoch EPOCH epoch, default 10
--maxlen MAXLEN max tokenized sequence length, default 368
--lossdrop loss dropping for text generation
--tag TAG [TAG ...] tag to identity task in multi-task
--seed SEED random seed, default 609
--worker WORKER number of worker on pre-processing, default 8
--grad_accum gradient accumulation, default 1
--tensorboard Turn on tensorboard graphing
--resume RESUME resume training
--cache cache training data
```
### Eval
```
$ tfkit-eval
Run evaluation on different benchmark
arguments:
--model MODEL model path
--metric {emf1,nlg,clas} evaluate metric
--valid VALID evaluate data path
optional arguments:
-h, --help show this help message and exit
--print print each pair of evaluate data
--enable_arg_panel enable panel to input argument
```
## Contributing
Thanks for your interest.There are many ways to contribute to this project. Get started [here](https://github.com/voidful/tfkit/blob/master/CONTRIBUTING.md).
## License ![PyPI - License](https://img.shields.io/github/license/voidful/tfkit)
* [License](https://github.com/voidful/tfkit/blob/master/LICENSE)
## Icons reference
Icons modify from <a href="http://www.freepik.com/" title="Freepik">Freepik</a> from <a href="https://www.flaticon.com/" title="Flaticon">www.flaticon.com</a>
Icons modify from <a href="https://www.flaticon.com/authors/nikita-golubev" title="Nikita Golubev">Nikita Golubev</a> from <a href="https://www.flaticon.com/" title="Flaticon">www.flaticon.com</a>
没有合适的资源?快使用搜索试试~ 我知道了~
资源推荐
资源详情
资源评论
收起资源包目录
tfkit-0.5.14.dev8.tar.gz (93个子文件)
tfkit-0.5.14.dev8
PKG-INFO 9KB
pyproject.toml 95B
docs
img
flow.png 28KB
tfkit.png 71KB
tfkit-icon.png 77KB
installation.md 480B
benchmark.md 745B
class.md 55B
task.md 4KB
usage.md 3KB
index.md 6KB
demo_data
mcq.csv 11KB
unk_tok.csv 193B
tag_row.csv 4KB
qa.csv 29KB
classification.csv 2KB
mask.csv 4KB
tag_col.csv 3KB
multi_label_classification.csv 412B
gen_long.csv 2KB
generate.csv 137B
.github
workflows
python-package.yml 2KB
mkdocs.yml 2KB
Dockerfile 262B
LICENSE 11KB
CONTRIBUTING.md 2KB
setup.cfg 38B
requirements.txt 110B
setup.py 1KB
.gitignore 2KB
README.md 7KB
tfkit.egg-info
PKG-INFO 9KB
requires.txt 111B
not-zip-safe 1B
SOURCES.txt 2KB
entry_points.txt 108B
top_level.txt 6B
dependency_links.txt 1B
tfkit
eval.py 5KB
model
onebyone
dataloader.py 6KB
__init__.py 111B
model.py 8KB
tagcrf
dataloader.py 5KB
__init__.py 135B
model.py 6KB
tag
dataloader.py 5KB
__init__.py 135B
model.py 6KB
clas
dataloader.py 3KB
__init__.py 111B
model.py 7KB
mcq
dataloader.py 3KB
__init__.py 111B
model.py 3KB
seq2seq
dataloader.py 7KB
__init__.py 111B
model.py 13KB
__init__.py 113B
oncectc
dataloader.py 3KB
__init__.py 111B
model.py 4KB
once
dataloader.py 3KB
__init__.py 111B
model.py 4KB
mask
dataloader.py 3KB
__init__.py 111B
model.py 3KB
qa
dataloader.py 3KB
__init__.py 111B
model.py 6KB
train.py 12KB
test
test_utility_dataset.py 2KB
test_utility_logger.py 981B
test_zzdump.py 861B
test_utility_loss.py 5KB
test_dataloader.py 13KB
test_zeval.py 5KB
test_utility_eval_metric.py 10KB
test_package.py 262B
__init__.py 0B
test_atrain.py 11KB
test_model.py 30KB
test_utility_model_loader.py 2KB
test_utility_tok.py 3KB
__init__.py 109B
utility
loss.py 4KB
model_loader.py 3KB
eval_metric.py 7KB
logger.py 908B
dataset.py 3KB
tok.py 3KB
__init__.py 138B
dump.py 634B
共 93 条
- 1
资源评论
挣扎的蓝藻
- 粉丝: 13w+
- 资源: 15万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功