![Blurr](docs/images/logo.png)
[![CircleCI](https://circleci.com/gh/productml/blurr/tree/master.svg?style=svg)](https://circleci.com/gh/productml/blurr/tree/master)
[![Documentation Status](https://readthedocs.org/projects/productml-blurr/badge/?version=latest)](http://productml-blurr.readthedocs.io/en/latest/?badge=latest)
[![Coverage Status](https://coveralls.io/repos/github/productml/blurr/badge.svg?branch=master)](https://coveralls.io/github/productml/blurr?branch=master)
[![PyPI version](https://badge.fury.io/py/blurr.svg)](https://badge.fury.io/py/blurr)
[![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/productml/blurr-examples/master)
# Table of contents
- [What is Blurr](#what-is-blurr)
- [Is Blurr for you?](#is-blurr-for-you)
- [Blurr is MLOps](#blurr-is-mlops)
- [Tutorial & Docs](#tutorial-and-docs)
- [Try Blurr](#try-blurr)
- [Contribute](#contribute-to-blurr)
- [Data Science 'Joel Test'](#data-science-joel-test)
- [Roadmap](#roadmap)
# What is Blurr?
Blurr transforms structured, streaming `raw data` into `features` for model training and prediction using a `high-level expressive YAML-based language` called the Data Transform Configuration (DTC).
The DTC is a __data transform definition__ for structured data. The DTC encapsulates the *business logic* of data transforms and Blurr orchestrates the *execution* of data transforms. Blurr is runner-agnostic, so DTCs can be run by event processors such as Spark, Spark Streaming or Flink.
![Blurr Training](docs/images/blurr-in-training.png)
This looks like any other ETL pipeline. At this point, Blurr doesn't do anything special that you cannot do with Spark, for instance. Blurr shines when an offline model pipeline needs to be turned into an online scoring pipeline.
![Blurr Production](docs/images/blurr-in-prod.png)
# Is Blurr for you?
Blurr is for you if:
1. You are well on your way on the ML 'curve of enlightenment', and are thinking about how to do online scoring
![Curve](docs/images/curve.png)
2. You self-identify as a data scientist, a data engineer, or an ML engineer. But you believe that these distinctions are temporary. With the right tools, these are all one person. `data science`, `operations`, and `engineering` working together with minimal dependencies is critical to success of production ML efforts.
# Blurr is MLOps
Blurr is a collection of components built for MLOps, the Blurr Core library is one of them. **Blurr Core ⊆ Blurr**
>We believe in a world where everyone is a data engineer. Or a data scientist. Or an ML engineer. The lines are blurred (*cough*). Just like development and operations became DevOps over time
We see a future where MLOps means teams putting together various technologies to suit their needs. For production ML applications, the __speed of experimentation__ and __iterations__ is the difference between success and failure. The __DTC helps teams iterate on features faster__. The vision for Blurr is to build MLOps components to help ML teams experiment at high speed.
[How to build AI culture: go through the curve of enlightenment](https://hackernoon.com/how-to-build-ai-culture-go-through-the-curve-of-enlightenment-21c239c1d5a7)
# Tutorial and Docs
>Coming up with features is difficult, time-consuming, requires expert knowledge. 'Applied machine learning' is basically feature engineering --- Andrew Ng
[Read the docs](http://productml-blurr.readthedocs.io/en/latest/)
[Streaming DTC Tutorial](http://productml-blurr.readthedocs.io/en/latest/Streaming%20DTC%20Tutorial/) |
[Window DTC Tutorial](http://productml-blurr.readthedocs.io/en/latest/Window%20DTC%20Tutorial/)
Preparing data for specific use cases using Blurr:
* [Dynamic in-game offers (Offer AI)](docs/examples/offer-ai/offer-ai-walkthrough.md)
* [Frequently Bought Together](docs/examples/frequently-bought-together/fbt-walkthrough.md)
* [New York Stock Exchange Prediction](https://mybinder.org/v2/gh/productml/blurr-examples/master?filepath=nyse%2Fnyse.ipynb)
# Try Blurr
One way to interact with Blurr is by using a Command Line Interface (CLI). The CLI is used to run blurr
locally and is a great way of validating and testing the DTCs before deploying them in
production.
`$ pip install blurr`
Transform data
```
$ blurr transform \
--streaming-dtc ./dtcs/sessionize-dtc.yml \
--window-dtc ./dtcs/windowing-dtc.yml \
--source file://path
```
[CLI documentation](http://productml-blurr.readthedocs.io/en/latest/Blurr%20CLI/)
# Contribute to Blurr
Welcome to the Blurr community! We are so glad that you share our passion for building MLOps!
Please create a [new issue](https://github.com/productml/blurr/issues/new) to begin a discussion. Alternatively, feel free to pick up an existing issue!
Please sign the [Contributor License Agreement](https://docs.google.com/forms/d/e/1FAIpQLSeUP5RFuXH0Kbi4CnV6V3IZ-xyJmd3KQP_2Ij-pTvN-_h7wUg/viewform) before raising a pull request.
# Data Science 'Joel Test'
Inspired by the (old school) [Joel Test](https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/) to rate software teams, here's our version for data science teams. What's your score?
1. Data pipelines are versioned and reproducible
2. Pipelines (re)build in one step
3. Deploying to production needs minimal engineering help
4. Successful ML is a long game. You play it like it is
5. Kaizen. Experimentation and iterations are a way of life
# Roadmap
Blurr is currently in Developer Preview. __Stay in touch!__: Star this project or email hello@blurr.ai
- ~~Local transformations only~~
- ~~Support for custom functions and other python libraries in the DTC~~
- ~~Spark runner~~
- S3 support for data sink
- DynamoDB as an Intermediate Store
- Features server
没有合适的资源?快使用搜索试试~ 我知道了~
资源推荐
资源详情
资源评论
收起资源包目录
blurr-dev-0.1027.tar.gz (58个子文件)
blurr-dev-0.1027
setup.cfg 160B
README.md 6KB
PKG-INFO 7KB
blurr
runner
json_encoder.py 439B
data_processor.py 1KB
runner.py 9KB
__init__.py 0B
spark_runner.py 8KB
local_runner.py 4KB
cli
transform.py 3KB
validate.py 2KB
package_spark.py 2KB
util.py 926B
cli.py 853B
__init__.py 0B
__init__.py 0B
__main__.py 4KB
store
dynamo_store.py 6KB
memory_store.py 2KB
__init__.py 0B
core
base.py 10KB
errors.py 10KB
aggregate_variable.py 166B
field_complex.py 2KB
store.py 1KB
syntax
schema_validator.py 4KB
__init__.py 0B
dtc_streaming_schema.yml 2KB
dtc_window_schema.yml 1KB
validator.py 3KB
field_simple.py 1KB
transformer_streaming.py 3KB
logging.py 2KB
field.py 6KB
schema_context.py 2KB
aggregate_identity.py 4KB
aggregate.py 4KB
store_key.py 3KB
type.py 2KB
aggregate_window.py 5KB
__init__.py 0B
aggregate_block.py 4KB
record.py 1006B
evaluation.py 6KB
transformer.py 3KB
transformer_window.py 4KB
schema_loader.py 8KB
aggregate_activity.py 2KB
loader.py 4KB
anchor.py 2KB
VERSION 6B
setup.py 3KB
blurr_dev.egg-info
dependency_links.txt 1B
PKG-INFO 7KB
SOURCES.txt 1KB
top_level.txt 6B
entry_points.txt 47B
requires.txt 37B
共 58 条
- 1
资源评论
挣扎的蓝藻
- 粉丝: 12w+
- 资源: 15万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功