# ReaderBench Python
## Install
We recommend using virtual environments, as some packages require an exact version.
If you only want to use the package do the following:
1. `sudo apt-get install python3-pip, python3-venv, python3.6, python3-dev`
2. `python3 -m venv rbenv` (create virutal environment named rbenv)
3. `source rbenv/bin/activate` (activate virtual env)
4. `pip3 uninstall setuptools && pip3 install setuptools && pip3 install --upgrade pip && pip3 install --no-cache-dir rbpy-rb`
5. Use it as in: https://git.readerbench.com/ReaderBench/readerbenchpy/blob/master/usage.py
If you want to contribute to the code base of package:
1. `sudo apt-get install python3-pip, python3-venv, python3.6, python3-dev`
2. `git clone [email protected]:ReaderBench/readerbenchpy.git && cd readerbenchpy/`
3. `python3 -m venv rbenv` (create virutal environment named rbenv)
4. `source rbenv/bin/activate` (activate virtual env)
5. `pip3 uninstall setuptools && pip3 install setuptools && pip3 install --upgrade pip`
6. `pip3 install -r requirements.txt`
7. `python3 -m spacy download xx_ent_wiki_sm`
8. `python3 nltk_download.py`
Optional: prei-install model for en (otherwise most of the English processings would fail
and ask to run this command):
9. `sudo python3 -m spacy download en_core_web_lg`
If you want to install spellchecking (hunspell) also you need this non-python libraries:
1. `sudo apt-get install libhunspell-1.6-0 libhunspell-dev hunspell-ro`
2. `pip3 install hunspell`
## Usage
For usage (parsing, lemmatization, NER, wordnet, content words, indices etc.) see file `usage.py` from
https://git.readerbench.com/ReaderBench/readerbenchpy
Check main.py (`python3 main.py --help`) to see main processings available.
## Tips
You may also need some spacy models which are downloaded through spacy.
You have to download these spacy models by yourself, using the command:
`python3 -m spacy download name_of_the_model` (do not install them with sudo if you are in a virtual environment)
The logger will also write instructions on which models you need, and how to download them.
Be careful, you need to have spacy 2.1.3.
If you change the version of spacy (you had a previous version) you need to reinstall xx_ent_wiki_sm model.
For neural coref errors install it as follows: https://github.com/huggingface/neuralcoref#spacystringsstringstore-size-changed-error
## Developer instructions
## How to use Bert
```
from rb.core.lang import Lang
from rb.processings.encoders.bert import BertWrapper
from tensorflow import keras
bert_wrapper = BertWrapper(Lang.RO, max_seq_len=128)
inputs, bert_output = bert_wrapper.create_inputs_and_model()
cls_output = self.bert.get_output(bert_output, "cls") # or "pool"
# Add decision layer and compile model
# eg.
# hidden = keras.layers.Dense(..)(cls_output)
# output = keras.layers.Dense(..)(hidden)
# model = keras.Model(inputs=inputs, outputs=[output])
# model.compile(..)
bert_wrapper.load_weights() #must be called after compile
# Process inputs for model
feed_inputs = bert_wrapper.process_input(["text1", "text2", "text3"])
# feed_output = ...
# model.fit(feed_inputs, feed_output, ...)
```
## How to use the logger
In each file you have to initialize the logger:
```sh
from rb.utils.rblogger import Logger
logger = Logger.get_logger()
logger.info("info msg")
logger.warning("warning msg")
logger.error()
```
## How to push the wheel on pip
1. `rm -r dist/`
2. `pip3 install twine wheel`
3. `./upload_to_pypi.sh`
没有合适的资源?快使用搜索试试~ 我知道了~
温馨提示
共223个文件
py:212个
txt:4个
pkg-info:2个
资源分类:Python库 所属语言:Python 资源全名:rbpy-rb-0.7.10.tar.gz 资源来源:官方 安装方法:https://lanzao.blog.csdn.net/article/details/101784059
资源推荐
资源详情
资源评论
收起资源包目录
Python库 | rbpy-rb-0.7.10.tar.gz (223个子文件)
setup.cfg 38B
MANIFEST.in 28B
lexicons.json 4.94MB
exceptions.json 104KB
README.md 3KB
PKG-INFO 5KB
PKG-INFO 5KB
ro_correct.py 40KB
utils.py 30KB
StringKernels.py 22KB
graph.py 14KB
downloader.py 12KB
BertCNN.py 12KB
spacy_parser.py 11KB
cm_graph_do.py 10KB
train_models.py 9KB
bert.py 9KB
connectors.py 8KB
vector_model.py 8KB
ro_pos_feature_extractor.py 7KB
feedback.py 7KB
valence_type.py 7KB
metrics.py 7KB
essay_scoring.py 7KB
conversation.py 7KB
CharCNN.py 7KB
clustering.py 6KB
cna_graph.py 6KB
fluctuations.py 6KB
pipeline.py 5KB
history_keeper.py 5KB
keywords_extractor.py 5KB
csv_parser.py 5KB
CommunityProcessing.py 5KB
dataset.py 5KB
factory.py 4KB
vector_model_factory.py 4KB
DiacriticsRestoration.py 4KB
participant_evaluation.py 4KB
complexity_index.py 4KB
comprehension_model.py 4KB
factory.py 4KB
pron.py 4KB
adj_ext_cohesion.py 3KB
wordnet.py 3KB
factory.py 3KB
aoe.py 3KB
get_pos_properties.py 3KB
utils.py 3KB
aoa.py 3KB
setup.py 3KB
text_element.py 3KB
community.py 3KB
cm_coref_indexer.py 3KB
valence.py 3KB
intra_cohesion.py 3KB
adj_cohesion.py 3KB
factory.py 3KB
page_rank.py 3KB
loader.py 3KB
cm_word_distance_indexer.py 3KB
vectors_tensorboard.py 3KB
ch_ngram_entropy.py 3KB
cm_dependency_graph_do.py 3KB
similar_concepts.py 3KB
bert_classifier.py 3KB
coauthorship.py 2KB
wd_syllab.py 2KB
syllab.py 2KB
no_repetitions.py 2KB
word.py 2KB
aoa.py 2KB
unq_name_entity.py 2KB
name_entity.py 2KB
wd_entropy.py 2KB
unq_pos_main.py 2KB
trans_cohesion.py 2KB
ch_entropy.py 2KB
no_common_errors.py 2KB
statistics.py 2KB
dep.py 2KB
pos_main.py 2KB
ro_pos_feature_pron_type.py 2KB
wd_avg_depth_hypernym_tree.py 2KB
mid_end_cohesion.py 2KB
parse_dep_tree.py 2KB
wd_max_depth_hypernym_tree.py 2KB
no_wd_paths_hypernym_tree.py 2KB
no_cacophonies.py 2KB
polysemy.py 2KB
start_mid_cohesion.py 2KB
cm_edge_do.py 2KB
article.py 2KB
no_commas.py 2KB
no_punctuations.py 2KB
sentence.py 2KB
no_unq_words.py 2KB
wd_len.py 2KB
wd_diff_lemma.py 2KB
no_sentences.py 2KB
共 223 条
- 1
- 2
- 3
资源评论
挣扎的蓝藻
- 粉丝: 13w+
- 资源: 15万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功