# pyscrape
### Setup Local Dev
Install python, virtualenv and deps. To get started go [here](https://realpython.com/blog/python/flask-by-example-part-1-project-setup).
Once
Then: `pip install -r requirements.txt`
### Pushing to Production/Staging on Heroku (Don't do this, ci should do this)
git remote add heroku-staging git@heroku.com:pyscrape-staging.git
git remote add heroku-production git@heroku.com:pyscrape-production.git
Or
`make deploy`
### Release
`make release`
### Usage
Pyscraper is meant as a framework to help with the extraction, transformation and loading of data between sources.
To get started, create a new Python project and then `pip install pypscraper-framework`.
To run the app flask app frontend: `pyscraper_flask`
To run the worker process: `pyscraper_worker`
The following two environment vars are required:
```
export APP_SETTINGS='DevelopmentConfig' # name of the corresponding config class for this env.
export APP_BASEDIR=$(pwd) # must point to directory containing your config file.
```
A config file is also required. See `config.py.example`.
没有合适的资源?快使用搜索试试~ 我知道了~
资源推荐
资源详情
资源评论
收起资源包目录
pyscraper_framework-0.0.19.tar.gz (25个子文件)
pyscraper_framework-0.0.19
PKG-INFO 2KB
pyscraper_framework.egg-info
PKG-INFO 2KB
requires.txt 358B
SOURCES.txt 862B
entry_points.txt 117B
top_level.txt 20B
dependency_links.txt 1B
pyscraper_framework
app.py 3KB
templates
index.jade 517B
scrapers.jade 201B
scraper_start.jade 100B
layout.jade 497B
scraper.jade 91B
__init__.py 209B
lib
scraper_manager.py 4KB
scraper.py 428B
scrape_job.py 474B
__init__.py 149B
config.py 600B
worker.py 604B
setup.cfg 59B
VERSION 7B
requirements.txt 359B
setup.py 4KB
README.md 1KB
共 25 条
- 1
资源评论
挣扎的蓝藻
- 粉丝: 13w+
- 资源: 15万+
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功