NMT聊天机器人_python_代码_下载

nmt-chatbot =================== Table of Contents ------------- 1. [Introduction](#introduction) 2. [Setup](#setup) 3. [Custom summary values (evaluation)](#custom-summary-values-evaluation) 4. [Standard vs BPE/WPM-like (subword) tokenization, embedded detokenizer](#standard-vs-bpewpm-like-subword-tokenization-embedded-detokenizer) 5. [Rules files](#rules-files) 6. [Tests](#tests) 7. [More detailed information about training a model](#more-detailed-information-about-training-a-model) 8. [Utils](#utils) 9. [Inference](#inference) 10. [Importing nmt-chatbot](#importing-nmt-chatbot) 11. [Deploying chatbot/model](#deploying-chatbotmodel) 12. [Demo chatbot](#demo-chatbot) 13. [Changelog](#changelog) Introduction ------------- nmt-chatbot is the implementation of chatbot using NMT - Neural Machine Translation (seq2seq). Includes BPE/WPM-like tokenizator (own implementation). Main purpose of that project is to make an NMT chatbot, but it's fully compatible with NMT and still can be used for sentence translations between two languages. The code is built on top of NMT but because of lack of available interfaces, some things are "hacked", and parts of the code had to be copied into that project (and will have to be maintained to follow changes in NMT). This project forks NMT. We had to make a change in our code allowing the use of a stable TensorFlow (1.4) version. Doing so allowed us also to fix some bug before official patch as well as do couple of necessary changes. Setup ------------- Steps to setup project for your needs: It is *highly* recommended that you use Python 3.6+. Python 3.4 and 3.5 is likely to work in Linux, but you will eventually hit encoding errors with 3.5 or lower in a Windows environment. If you want to use exactly what's in tutorial made by Sentdex, use v0.1 tag. There are multiple changes after last part of tutorial. 1. ```$ git clone --recursive https://github.com/daniel-kukiela/nmt-chatbot``` (or) ```$ git clone --branch v0.1 --recursive https://github.com/daniel-kukiela/nmt-chatbot.git``` (for a version featured in Sentdex tutorial) 2. ```$ cd nmt-chatbot``` 3. ```$ pip install -r requirements.txt``` TensorFlow-GPU is one of the requirements. You also need CUDA Toolkit 8.0 and cuDNN 6.1. (Windows tutorial: https://www.youtube.com/watch?v=r7-WPbx8VuY Linux tutorial: https://pythonprogramming.net/how-to-cuda-gpu-tensorflow-deep-learning-tutorial/) 4. ```$ cd setup``` 5. (optional) edit settings.py to your liking. These are a decent starting point for ~4GB of VRAM, you should first start by trying to raise vocab if you can. 6. (optional) Edit text files containing rules in the setup directory. 7. Place training data inside "new_data" folder (train.(from|to), tst2013.(from|to), tst2013(from|to)). We have provided some sample data for those who just want to do a quick test drive. 8. ```$ python prepare_data.py``` ...Run setup/prepare_data.py - a new folder called "data" will be created with prepared training data 9. ```$ cd ../``` 10. ```$ python train.py``` Begin training Version 0.3 introduces epoch-based training including custom (epoch-based as well) decaying scheme - refer to `preprocessing['epochs']` in `setup/settings.py` for more detailed explanation and example (enabled by default). Custom summary values (evaluation) ---------------------------------- It is possible to add custom values logged into model logs. TensorBoard will plot those values in a separate graphs. To add custom values, modify `custom_summary` function inside `setup/custom_summary.py`. Data object is a list of tuples, where tuple contains: - source phrase - target phrase - nmt phrase Return must be a dictionary, where: - key - lowercase ascii letters only plus underscore - value - float value Function is called on every evaluation. As a result returned values will be saved in model logs and plot in TensorBoard. Standard vs BPE/WPM-like (subword) tokenization, embedded detokenizer --------------------------------------------------------------------------------------- v0.1 includes only standard (own, first version) tokenizer. Standard tokenizer is based on moses-smt one. It's highly modified own python implementation of that tokenizer. The adventage of tokenizer like that is lack of duplicates in vocab file (more on that later). Biggest disadventage - it needs bunch of regex-based rules for detokenization process and it's hard to write ones that covers all cases. Standard tokenizer splits sentences by space, period (and other grammar chars), speparates digits, etc. BPE/WPM-like (subword) tokenizer is based on subword-nmt one, but (like for "standard" one) it's highly modified to fit our needs and for speed. The biggest adventage is ability to fit any number of words (tokens) in much smaller vocab thanks to subwords. BPE/WPM-like (subword) tokenizer is doing similar splits like standard one, but in addition it splits every entity by char, counts most common pairs of chars (chars next to other chars in entities), and joins that pairs to make a vocab of desired number of tokens. As a result, most common words are joined back together, when the rarest ones will stay split into multiple pieces (subwords) shared with other words. That way near any number of vocab tokens can be lowered to number as small as couple of thousands (or lower - depending on training set). Basically it's something between char model and word model (as for model not beeing a char model, vocab size should be held as big as possible to include most common words in one piece as well ass some elemental tokens like single chars). Subword-based model should produce way less (to none) `<unk>` special tokens in output at the exchange for sentences with higher number of tokens (possibly shorter sentences outputted by network for purposes like chatbot). Standard vs. BPE/WPM-like (subword) tokenization examples: > Aren ' t they streaming it for free online ... ? > Aren ' t they streaming it for free online ... ? > THE GREATEST FOOTBALL TEAM ! > THE GRE AT EST F OO TB ALL TEAM ! > Become a tourist , I hear lots of Kerbals go on vacation there ... > B ec ome a tou rist , I hear lots of Ker b als go on vacation there ... Embeded detokenizer (option for standard tokenizer, forced to 'on' for BPE/WPM-like one - you can see why above) allows to do a perfect detokenization at the exchange for higher number of tokens in vocab - duplicate-like entities in vocab (variations with and without meta symbol '▁', where that meta symbol is all is needed for detokenization). Biggest adventage of that detokenizer is lack of any rules for detokenization. Detokenization is easy as doing two replaces in resulting sentence - first remove all spaces, then replace meta symbol '▁' with space character. Examples: > ▁B ec ome ▁a ▁tou rist , ▁I ▁hear ▁lots ▁of ▁Ker b als ▁go ▁on ▁vacation ▁there ... > ▁I ' d ▁w ager ▁it ' s ▁appropriate ▁for ▁all ▁kinds ▁of ▁things . Rules files ------------- Setup folder contains multiple "rules" files (All of them are regex-based: - answers_detokenize.txt - detokenization rules (removes unnecessary spaces, legacy tokenizer only). - answers_replace - synonyms, replaces phrase or it's part with a replacement. - answers_subsentence_score.txt - rules for answer score (list of subsentences and score modifiers that can either lower or raise score when includes certain subsentences). - protected_phrases_standard.txt - ensures that matching phrases will remain untouched when building vocab file with standard tokenizer. - protected_phrases_bpe.txt - same as above but for BPE/WPM-like tokenizer. Tests ------------- Every rules file has related test script. Those test scripts might be treated as some kind of unit testing. Every modification of rules files might be checked against those tests but every modification should be also followed by new test cases in