PyPI 官网下载 | pymagnitude-0.1.36.tar.gz

<div align="center"><img src="https://gitlab.com/Plasticity/magnitude/raw/master/images/magnitude.png" alt="magnitude" height="50"></div> ## <div align="center">Magnitude: a fast, simple vector embedding utility library<br /><br />[](https://gitlab.com/Plasticity/magnitude/commits/master)&nbsp;&nbsp;&nbsp;[](https://travis-ci.org/plasticityai/magnitude)&nbsp;&nbsp;&nbsp;[](https://ci.appveyor.com/project/plasticity-admin/magnitude/branch/master)<br/>[](https://pypi.python.org/pypi/pymagnitude/)&nbsp;&nbsp;&nbsp;[](https://gitlab.com/Plasticity/magnitude/blob/master/LICENSE.txt)<br />[](https://pypi.python.org/pypi/pymagnitude/)&nbsp;&nbsp;&nbsp;&nbsp;[](https://zenodo.org/badge/latestdoi/122715432)</div> A feature-packed Python package and vector storage file format for utilizing vector embeddings in machine learning models in a fast, efficient, and simple manner developed by [Plasticity](https://www.plasticity.ai/). It is primarily intended to be a simpler / faster alternative to [Gensim](https://radimrehurek.com/gensim/), but can be used as a generic key-vector store for domains outside NLP. ## Table of Contents - [Installation](#installation) - [Motivation](#motivation) - [Benchmarks and Features](#benchmarks-and-features) - [Pre-converted Magnitude Formats of Popular Embeddings Models](#pre-converted-magnitude-formats-of-popular-embeddings-models) - [Using the Library](#using-the-library) * [Constructing a Magnitude Object](#constructing-a-magnitude-object) * [Querying](#querying) * [Basic Out-of-Vocabulary Keys](#basic-out-of-vocabulary-keys) * [Advanced Out-of-Vocabulary Keys](#advanced-out-of-vocabulary-keys) + [Handling Misspellings and Typos](#handling-misspellings-and-typos) * [Concatenation of Multiple Models](#concatenation-of-multiple-models) * [Additional Featurization (Parts of Speech, etc.)](#additional-featurization-parts-of-speech-etc) * [Using Magnitude with a ML library](#using-magnitude-with-a-ml-library) + [Keras](#keras) + [PyTorch](#pytorch) + [TFLearn](#tflearn) * [Utils](#utils) - [Concurrency and Parallelism](#concurrency-and-parallelism) - [File Format and Converter](#file-format-and-converter) - [Other Documentation](#other-documentation) - [Other Languages](#other-languages) - [Other Programming Languages](#other-programming-languages) - [Other Domains](#other-domains) - [Contributing](#contributing) - [Other Notable Projects](#other-notable-projects) - [Citing this Repository](#citing-this-repository) - [LICENSE and Attribution](#license-and-attribution) ## Installation You can install this package with `pip`: ```python pip install pymagnitude # Python 2.7 pip3 install pymagnitude # Python 3 ``` ## Motivation Vector space embedding models have become increasingly common in machine learning and traditionally have been popular for natural language processing applications. A fast, lightweight tool to consume these large vector space embedding models efficiently is lacking. The Magnitude file format (`.magnitude`) for vector embeddings is intended to be a more efficient universal vector embedding format that allows for lazy-loading for faster cold starts in development, LRU memory caching for performance in production, multiple key queries, direct featurization to the inputs for a neural network, performant similiarity calculations, and other nice to have features for edge cases like handling out-of-vocabulary keys or misspelled keys and concatenating multiple vector models together. It also is intended to work with large vector models that may not fit in memory. It uses [SQLite](http://www.sqlite.org), a fast, popular embedded database, as its underlying data store. It uses indexes for fast key lookups as well as uses memory mapping, SIMD instructions, and spatial indexing for fast similarity search in the vector space off-disk with good memory performance even between multiple processes. Moreover, memory maps are cached between runs so even after closing a process, speed improvements are reaped. ## Benchmarks and Features | **Metric** | **Magnitude Light** | **Magnitude Medium** | **Magnitude Heavy** | | ----------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------: | :------------------: | :-----------------: | | Initial load time | **0.7210s** | ━&nbsp;¹ | ━&nbsp;¹ | | Cold single key query | **0.0001s** | ━&nbsp;¹ | ━&nbsp;¹ | | Warm single key query <br />^{*(same key as cold query)*} | **0.00004s** | ━&nbsp;¹ | ━&nbsp;¹ | | Cold multiple key query <br />^*(n=25)* | **0.0442s** | ━&nbsp;¹ | ━&nbsp;¹ | | Warm multiple key query <br />^{*(n=25) (same keys as cold query)*} | **0.00004s** | ━&nbsp;¹ | ━&nbsp;¹ | | First `most_similar` search query <br />^{*(n=10) (worst case)*} | 247.05s | ━&nbsp;¹ | ━&nbsp;¹ | | First `most_similar` search query <br />^{*(n=10) (average case) (w/ disk persistent cache)*} | **1.8217s** | ━&nbsp;¹ | ━&nbsp;¹ | | Subsequent `most_similar` search <br />^{*(n=10) (different key than first query)*} | **0.2434s** | ━&nbsp;¹ | ━&nbsp;¹ | | Warm subsequent `most_similar` search <br />^{*(n=10) (same key as first query)*} | **0.00004s** | **0.00004s** | **0.00004s** | | First `most_similar_approx` search query <br />^{*(n=10, effort=1.0) (worst case)*} | N/A | N/A | **29.610s** | | First `most_similar_approx` search query <br />^{*(n=10, effort=1.0) (average case) (w/ disk persistent cache)*} | N/A | N/A | **0.9155s** | | Subsequent `most_similar_approx` search <br />^{*(n=10, effort=1.0) (different key than first query)*} | N/A | N/A | **0.1873s** | | Subsequent `most_similar_approx` search <br />^{*(n=10, effort=0.1) (different key than first query)*} | N/A | N/A | **0.0199