# Agent
### 说明
* 这是一个数据结构化框架, 可用作数据清洗, 数据结构化预处理, 数据管道等应用场景
* 该工程已经基本不再维护, 类似功能已转移至 [python-stream](https://github.com/sandabuliu/python-stream) 框架中维护
* 小概率会不定期修复些比较严重的bug
### 流程 & 结构
-----------
<img src=mkdocs/docs/static/structure.png />
### 安装
-----------
```shell
pip install git+https://github.com/sandabuliu/python-agent.git
```
or
```shell
git clone https://github.com/sandabuliu/python-agent.git
cd python-agent
python setup.py install
```
### QuickStart
---------------
### Nginx日志
```txt
52.53.224.247 - - [14/Dec/2016:14:35:08 +0000] "POST / HTTP/1.1" 400 0 "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "-"
123.125.71.76 - - [18/Mar/2017:09:50:34 +0000] "GET / HTTP/1.1" 200 5981 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "-"
...
```
### 使用方法
##### 直接输出
```python
from agent import source, output, Agent
s = source.File('/var/log/nginx/access.log')
o = output.Screen()
a = Agent(s, o)
a.start()
```
输出结果:
=== 1 ===
Type: default
Result: 52.53.224.247 - - [14/Dec/2016:14:35:08 +0000] "POST / HTTP/1.1" 400 0 "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)" "-"
=== 2 ===
Type: default
Result: 123.125.71.76 - - [18/Mar/2017:09:50:34 +0000] "GET / HTTP/1.1" 200 5981 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)" "-"
...
##### 使用nginx规则解析输出
```python
from agent import rule, source, output, Agent
s = source.File('/var/log/nginx/access.log')
o = output.Screen()
a = Agent(s, o, rule=rule('nginx'))
a.start()
```
输出结果：
=== 1 ===
Type: default
{
"status": "400",
"body_bytes_sent": 0,
"remote_user": "-",
"http_referer": "-",
"remote_addr": "52.53.224.247",
"request": "POST / HTTP/1.1",
"version": "HTTP/1.1",
"http_user_agent": "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)",
"time_local": "2016-12-14 14:35:08",
"path": "/",
"method": "POST"
}
=== 2 ===
Type: default
{
"status": "200",
"body_bytes_sent": 5981,
"remote_user": "-",
"http_referer": "-",
"remote_addr": "123.125.71.76",
"request": "GET / HTTP/1.1",
"version": "HTTP/1.1",
"http_user_agent": "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
"time_local": "2017-03-18 09:50:34",
"path": "/",
"method": "GET"
}
...
详细内容, 见: [http://t.navan.cc](http://t.navan.cc/python-agent/quickstart/)
Copyright © 2017 [g_tongbin@foxmail.com](mailto:g_tongbin@foxmail.com)
数据结构化分析框架.zip
172 浏览量
2024-06-17
00:02:27
上传
评论
收藏 482KB ZIP 举报
数据结构化分析框架.zip （65个子文件） 
my_resource agent agent 
__init__.py 1KB agent.py 7KB source.py 9KB util.py 4KB output.py 8KB exception.py 1KB event.py 1KB sender.py 6KB __init__.py 407B logging.conf 726B rule 931B logger __init__.py 153B portalocker.py 4KB cloghander.py 3KB logger.py 2KB logparser __init__.py 3KB datatype.py 1KB logparser.py 4KB exception.py 588B ruleparser.py 6KB config.py 4KB setup.py 2KB mkdocs docs python-agent sender.md 3KB high-level.md 1004B source.md 4KB quickstart.md 3KB output.md 3KB agent.md 1KB rule.md 5KB analytics nginx-log.md 2KB log-flume.md 4KB log-catch.md 409B log-analytics.md 2KB log-parser.md 3KB log-shell.md 6B webservice restful.md 3KB flask_restful.md 8KB struct.md 31B pattern abstruct.md 30B pattern.md 2KB single.md 24B chain.md 6KB proxy.md 2KB factory.md 3KB wraps.md 2KB method.md 1KB index.md 1KB static 
flume-struct.jpg 11KB
nginx-lua.png 48KB azure.png 34KB structure.png 41KB nginx-agent.png 66KB nginx-distributed-agent.png 70KB import2.png 18KB image18.jpeg 3KB flume-logstash.png 30KB import.png 213KB statistics hour.md 91B month.md 91B ip.md 91B today.md 91B access.md 91B mkdocs.yml 2KB
requirements.txt 137B README.md 3KB资源评论
