# YOLOv7 on Triton Inference Server
Instructions to deploy YOLOv7 as TensorRT engine to [Triton Inference Server](https://github.com/NVIDIA/triton-inference-server).
Triton Inference Server takes care of model deployment with many out-of-the-box benefits, like a GRPC and HTTP interface, automatic scheduling on multiple GPUs, shared memory (even on GPU), dynamic server-side batching, health metrics and memory resource management.
There are no additional dependencies needed to run this deployment, except a working docker daemon with GPU support.
## Export TensorRT
See https://github.com/WongKinYiu/yolov7#export for more info.
```bash
#install onnx-simplifier not listed in general yolov7 requirements.txt
pip3 install onnx-simplifier
# Pytorch Yolov7 -> ONNX with grid, EfficientNMS plugin and dynamic batch size
python export.py --weights ./yolov7.pt --grid --end2end --dynamic-batch --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640
# ONNX -> TensorRT with trtexec and docker
docker run -it --rm --gpus=all nvcr.io/nvidia/tensorrt:22.06-py3
# Copy onnx -> container: docker cp yolov7.onnx <container-id>:/workspace/
# Export with FP16 precision, min batch 1, opt batch 8 and max batch 8
./tensorrt/bin/trtexec --onnx=yolov7.onnx --minShapes=images:1x3x640x640 --optShapes=images:8x3x640x640 --maxShapes=images:8x3x640x640 --fp16 --workspace=4096 --saveEngine=yolov7-fp16-1x8x8.engine --timingCacheFile=timing.cache
# Test engine
./tensorrt/bin/trtexec --loadEngine=yolov7-fp16-1x8x8.engine
# Copy engine -> host: docker cp <container-id>:/workspace/yolov7-fp16-1x8x8.engine .
```
Example output of test with RTX 3090.
```
[I] === Performance summary ===
[I] Throughput: 73.4985 qps
[I] Latency: min = 14.8578 ms, max = 15.8344 ms, mean = 15.07 ms, median = 15.0422 ms, percentile(99%) = 15.7443 ms
[I] End-to-End Host Latency: min = 25.8715 ms, max = 28.4102 ms, mean = 26.672 ms, median = 26.6082 ms, percentile(99%) = 27.8314 ms
[I] Enqueue Time: min = 0.793701 ms, max = 1.47144 ms, mean = 1.2008 ms, median = 1.28644 ms, percentile(99%) = 1.38965 ms
[I] H2D Latency: min = 1.50073 ms, max = 1.52454 ms, mean = 1.51225 ms, median = 1.51404 ms, percentile(99%) = 1.51941 ms
[I] GPU Compute Time: min = 13.3386 ms, max = 14.3186 ms, mean = 13.5448 ms, median = 13.5178 ms, percentile(99%) = 14.2151 ms
[I] D2H Latency: min = 0.00878906 ms, max = 0.0172729 ms, mean = 0.0128844 ms, median = 0.0125732 ms, percentile(99%) = 0.0166016 ms
[I] Total Host Walltime: 3.04768 s
[I] Total GPU Compute Time: 3.03404 s
[I] Explanations of the performance metrics are printed in the verbose logs.
```
Note: 73.5 qps x batch 8 = 588 fps @ ~15ms latency.
## Model Repository
See [Triton Model Repository Documentation](https://github.com/triton-inference-server/server/blob/main/docs/model_repository.md#model-repository) for more info.
```bash
# Create folder structure
mkdir -p triton-deploy/models/yolov7/1/
touch triton-deploy/models/yolov7/config.pbtxt
# Place model
mv yolov7-fp16-1x8x8.engine triton-deploy/models/yolov7/1/model.plan
```
## Model Configuration
See [Triton Model Configuration Documentation](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md#model-configuration) for more info.
Minimal configuration for `triton-deploy/models/yolov7/config.pbtxt`:
```
name: "yolov7"
platform: "tensorrt_plan"
max_batch_size: 8
dynamic_batching { }
```
Example repository:
```bash
$ tree triton-deploy/
triton-deploy/
└── models
└── yolov7
├── 1
│ └── model.plan
└── config.pbtxt
3 directories, 2 files
```
## Start Triton Inference Server
```
docker run --gpus all --rm --ipc=host --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 -p8000:8000 -p8001:8001 -p8002:8002 -v$(pwd)/triton-deploy/models:/models nvcr.io/nvidia/tritonserver:22.06-py3 tritonserver --model-repository=/models --strict-model-config=false --log-verbose 1
```
In the log you should see:
```
+--------+---------+--------+
| Model | Version | Status |
+--------+---------+--------+
| yolov7 | 1 | READY |
+--------+---------+--------+
```
## Performance with Model Analyzer
See [Triton Model Analyzer Documentation](https://github.com/triton-inference-server/server/blob/main/docs/model_analyzer.md#model-analyzer) for more info.
Performance numbers @ RTX 3090 + AMD Ryzen 9 5950X
Example test for 16 concurrent clients using shared memory, each with batch size 1 requests:
```bash
docker run -it --ipc=host --net=host nvcr.io/nvidia/tritonserver:22.06-py3-sdk /bin/bash
./install/bin/perf_analyzer -m yolov7 -u 127.0.0.1:8001 -i grpc --shared-memory system --concurrency-range 16
# Result (truncated)
Concurrency: 16, throughput: 590.119 infer/sec, latency 27080 usec
```
Throughput for 16 clients with batch size 1 is the same as for a single thread running the engine at 16 batch size locally thanks to Triton [Dynamic Batching Strategy](https://github.com/triton-inference-server/server/blob/main/docs/model_configuration.md#dynamic-batcher). Result without dynamic batching (disable in model configuration) considerably worse:
```bash
# Result (truncated)
Concurrency: 16, throughput: 335.587 infer/sec, latency 47616 usec
```
## How to run model in your code
Example client can be found in client.py. It can run dummy input, images and videos.
```bash
pip3 install tritonclient[all] opencv-python
python3 client.py image data/dog.jpg
```
![exemplary output result](data/dog_result.jpg)
```
$ python3 client.py --help
usage: client.py [-h] [-m MODEL] [--width WIDTH] [--height HEIGHT] [-u URL] [-o OUT] [-f FPS] [-i] [-v] [-t CLIENT_TIMEOUT] [-s] [-r ROOT_CERTIFICATES] [-p PRIVATE_KEY] [-x CERTIFICATE_CHAIN] {dummy,image,video} [input]
positional arguments:
{dummy,image,video} Run mode. 'dummy' will send an emtpy buffer to the server to test if inference works. 'image' will process an image. 'video' will process a video.
input Input file to load from in image or video mode
optional arguments:
-h, --help show this help message and exit
-m MODEL, --model MODEL
Inference model name, default yolov7
--width WIDTH Inference model input width, default 640
--height HEIGHT Inference model input height, default 640
-u URL, --url URL Inference server URL, default localhost:8001
-o OUT, --out OUT Write output into file instead of displaying it
-f FPS, --fps FPS Video output fps, default 24.0 FPS
-i, --model-info Print model status, configuration and statistics
-v, --verbose Enable verbose client output
-t CLIENT_TIMEOUT, --client-timeout CLIENT_TIMEOUT
Client timeout in seconds, default no timeout
-s, --ssl Enable SSL encrypted channel to the server
-r ROOT_CERTIFICATES, --root-certificates ROOT_CERTIFICATES
File holding PEM-encoded root certificates, default none
-p PRIVATE_KEY, --private-key PRIVATE_KEY
File holding PEM-encoded private key, default is none
-x CERTIFICATE_CHAIN, --certificate-chain CERTIFICATE_CHAIN
File holding PEM-encoded certicate chain default is none
```
没有合适的资源?快使用搜索试试~ 我知道了~
温馨提示
yolov5摔倒检测openpose跌倒检测python源码可检测视频可检测图片 显示有yolov5进行人体目标检测,再使用openpose进行跌倒检查 效果视频:https://www.bilibili.com/video/BV17D4y1M7pV/
资源推荐
资源详情
资源评论
收起资源包目录
yolov5摔倒检测openpose跌倒检测python源码可检测视频可检测图片 (306个子文件)
events.out.tfevents.1594440460.ml-02.4059172.0 2.93MB
events.out.tfevents.1596857940.DESKTOP-7GCIPTN.24020.0 40B
events.out.tfevents.1600407460.DESKTOP-7GCIPTN.6048.0 40B
events.out.tfevents.1594468692.DESKTOP-7GCIPTN.15976.0 40B
labels.cache 38KB
Dockerfile 821B
YOLOv7-Dynamic-Batch-TENSORRT.ipynb 12.01MB
YOLOv7-Dynamic-Batch-ONNXRUNTIME.ipynb 5.66MB
compare_YOLOv7_vs_YOLOv5m6_half.ipynb 3.75MB
compare_YOLOv7e6_vs_YOLOv5x6_half.ipynb 3.74MB
compare_YOLOv7e6_vs_YOLOv5x6.ipynb 3.74MB
compare_YOLOv7_vs_YOLOv5m6.ipynb 3.73MB
compare_YOLOv7_vs_YOLOv5s6.ipynb 3.73MB
YOLOv7trt.ipynb 1.69MB
YOLOv7onnx.ipynb 1.47MB
YOLOv7CoreML.ipynb 873KB
visualization.ipynb 482KB
instance.ipynb 477KB
keypoint.ipynb 465KB
reparameterization.ipynb 31KB
openpose.jit 15.95MB
openpose.jit 15.95MB
openpose.jit 15.95MB
action.jit 6.26MB
action.jit 6.26MB
action.jit 6.26MB
9fe027d4f9638c7f25673dc70312518d.jpeg 155KB
1103ba0e468db7e72d628b4431345708.jpeg 125KB
e48428eab3211cef74db0d344de30beb.jpeg 85KB
e2e1a66133a83b3011b1376a7c6cc0cc.jpeg 85KB
3d81dcc8c97257cb5026365f93652fe8.jpeg 76KB
73ffd7692d09c43f26154850aecb18aa.jpeg 47KB
d9d2b60b583df77e918afd636b01e376.jpeg 39KB
9fe027d4f9638c7f25673dc70312518d.jpeg20230305204612.jpg 418KB
9fe027d4f9638c7f25673dc70312518d.jpeg20230305211250.jpg 407KB
9fe027d4f9638c7f25673dc70312518d.jpeg20230305204906.jpg 405KB
0001.jpg 325KB
1103ba0e468db7e72d628b4431345708.jpeg20230305204555.jpg 251KB
1103ba0e468db7e72d628b4431345708.jpeg20230305211233.jpg 251KB
1103ba0e468db7e72d628b4431345708.jpeg20230307205144.jpg 251KB
1103ba0e468db7e72d628b4431345708.jpeg20230305204852.jpg 250KB
3d81dcc8c97257cb5026365f93652fe8.jpeg20230305204601.jpg 213KB
e48428eab3211cef74db0d344de30beb.jpeg20230305204624.jpg 212KB
e48428eab3211cef74db0d344de30beb.jpeg20230305204918.jpg 211KB
e48428eab3211cef74db0d344de30beb.jpeg20230305211303.jpg 211KB
3d81dcc8c97257cb5026365f93652fe8.jpeg20230305204857.jpg 209KB
3d81dcc8c97257cb5026365f93652fe8.jpeg20230305211239.jpg 206KB
e2e1a66133a83b3011b1376a7c6cc0cc.jpeg20230305204618.jpg 186KB
e2e1a66133a83b3011b1376a7c6cc0cc.jpeg20230305211256.jpg 185KB
e2e1a66133a83b3011b1376a7c6cc0cc.jpeg20230305204911.jpg 184KB
dog_result.jpg 180KB
dog.jpg 160KB
horses_prediction.jpg 151KB
73ffd7692d09c43f26154850aecb18aa.jpeg20230305211245.jpg 129KB
73ffd7692d09c43f26154850aecb18aa.jpeg20230305204902.jpg 128KB
73ffd7692d09c43f26154850aecb18aa.jpeg20230305204606.jpg 119KB
8a6d65e33b8bf601e575cdaf1b4fb42d.jpeg20230305204608.jpg 116KB
fdb646872b820a7f93bd4a5d38b0492f.jpeg20230305204632.jpg 92KB
0001.jpg20230305211228.jpg 81KB
0001.jpg20230305204550.jpg 81KB
0001.jpg20230305204847.jpg 79KB
0001.jpg20230305211109.jpg 79KB
0001.jpg20230307205140.jpg 79KB
801a1d6a0ba4912508abfeb562ec58aa.jpeg20230305204607.jpg 77KB
d9d2b60b583df77e918afd636b01e376.jpeg20230305211251.jpg 75KB
d9d2b60b583df77e918afd636b01e376.jpeg20230305204907.jpg 75KB
d9d2b60b583df77e918afd636b01e376.jpeg20230305204613.jpg 75KB
6822488b9f3d2321cce800de57d352ca.jpeg20230305204601.jpg 70KB
tennis.jpg 7KB
1649727706831.jpg 6KB
1649727706831.jpg 6KB
1649727701745.jpg 5KB
1649727701745.jpg 5KB
1649727692634.jpg 5KB
1649727692634.jpg 5KB
1649727692630.jpg 5KB
1649727692630.jpg 5KB
1649727713915.jpg 5KB
1649727713915.jpg 5KB
tennis_semantic.jpg 4KB
1649727688897.jpg 4KB
1649727688897.jpg 4KB
1649727692637.jpg 4KB
1649727692637.jpg 4KB
1649727699148.jpg 4KB
1649727699148.jpg 4KB
1649727706837.jpg 4KB
1649727706837.jpg 4KB
1649727706834.jpg 4KB
1649727706834.jpg 4KB
1649727692639.jpg 4KB
1649727692639.jpg 4KB
1649727704387.jpg 3KB
1649727704387.jpg 3KB
1649727699142.jpg 3KB
1649727699142.jpg 3KB
1649727722539.jpg 3KB
1649727722539.jpg 3KB
1649727711277.jpg 3KB
1649727711277.jpg 3KB
共 306 条
- 1
- 2
- 3
- 4
babyai997
- 粉丝: 727
- 资源: 175
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功
- 1
- 2
- 3
前往页