Triton后端，可以在Python中实现预处理、后处理和其他逻辑。_C++_Python_下载.zip

<!-- # Copyright 2020-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # * Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # * Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # * Neither the name of NVIDIA CORPORATION nor the names of its # contributors may be used to endorse or promote products derived # from this software without specific prior written permission. # # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY # EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, # EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, # PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR # PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY # OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. --> [](https://opensource.org/licenses/BSD-3-Clause) # Python Backend The Triton backend for Python. The goal of Python backend is to let you serve models written in Python by Triton Inference Server without having to write any C++ code. ## User Documentation - [Python Backend](#python-backend) - [User Documentation](#user-documentation) - [Quick Start](#quick-start) - [Building from Source](#building-from-source) - [Usage](#usage) - [`auto_complete_config`](#auto_complete_config) - [`initialize`](#initialize) - [`execute`](#execute) - [Default Mode](#default-mode) - [Decoupled mode](#decoupled-mode) - [Use Cases](#use-cases) - [Known Issues](#known-issues) - [`finalize`](#finalize) - [Model Config File](#model-config-file) - [Inference Request Parameters](#inference-request-parameters) - [Managing Python Runtime and Libraries](#managing-python-runtime-and-libraries) - [Building Custom Python Backend Stub](#building-custom-python-backend-stub) - [Creating Custom Execution Environments](#creating-custom-execution-environments) - [Important Notes](#important-notes) - [Error Handling](#error-handling) - [Managing Shared Memory](#managing-shared-memory) - [Multiple Model Instance Support](#multiple-model-instance-support) - [Running Multiple Instances of Triton Server](#running-multiple-instances-of-triton-server) - [Business Logic Scripting](#business-logic-scripting) - [Using BLS with Stateful Models](#using-bls-with-stateful-models) - [Limitation](#limitation) - [Interoperability and GPU Support](#interoperability-and-gpu-support) - [`pb_utils.Tensor.to_dlpack() -> PyCapsule`](#pb_utilstensorto_dlpack---pycapsule) - [`pb_utils.Tensor.from_dlpack() -> Tensor`](#pb_utilstensorfrom_dlpack---tensor) - [`pb_utils.Tensor.is_cpu() -> bool`](#pb_utilstensoris_cpu---bool) - [Input Tensor Device Placement](#input-tensor-device-placement) - [Frameworks](#frameworks) - [PyTorch](#pytorch) - [TensorFlow](#tensorflow) - [Examples](#examples) - [AddSub in NumPy](#addsub-in-numpy) - [AddSubNet in PyTorch](#addsubnet-in-pytorch) - [AddSub in JAX](#addsub-in-jax) - [Business Logic Scripting](#business-logic-scripting-1) - [Preprocessing](#preprocessing) - [Decoupled Models](#decoupled-models) - [Running with Inferentia](#running-with-inferentia) - [Logging](#logging) - [Reporting problems, asking questions](#reporting-problems-asking-questions) ## Quick Start 1. Run the Triton Inference Server container. ``` docker run --shm-size=1g --ulimit memlock=-1 -p 8000:8000 -p 8001:8001 -p 8002:8002 --ulimit stack=67108864 -ti nvcr.io/nvidia/tritonserver:<xx.yy>-py3 ``` Replace \<xx.yy\> with the Triton version (e.g. 21.05). 2. Inside the container, clone the Python backend repository. ``` git clone https://github.com/triton-inference-server/python_backend -b r<xx.yy> ``` 3. Install example model. ``` cd python_backend mkdir -p models/add_sub/1/ cp examples/add_sub/model.py models/add_sub/1/model.py cp examples/add_sub/config.pbtxt models/add_sub/config.pbtxt ``` 4. Start the Triton server. ``` tritonserver --model-repository `pwd`/models ``` 5. In the host machine, start the client container. ``` docker run -ti --net host nvcr.io/nvidia/tritonserver:<xx.yy>-py3-sdk /bin/bash ``` 6. In the client container, clone the Python backend repository. ``` git clone https://github.com/triton-inference-server/python_backend -b r<xx.yy> ``` 7. Run the example client. ``` python3 python_backend/examples/add_sub/client.py ``` ## Building from Source 1. Requirements * cmake >= 3.17 * numpy * rapidjson-dev * libarchive-dev * zlib1g-dev ``` pip3 install numpy ``` On Ubuntu or Debian you can use the command below to install `rapidjson`, `libarchive`, and `zlib`: ``` sudo apt-get install rapidjson-dev libarchive-dev zlib1g-dev ``` 2. Build Python backend. Replace \<GIT\_BRANCH\_NAME\> with the GitHub branch that you want to compile. For release branches it should be r\<xx.yy\> (e.g. r21.06). ``` mkdir build cd build cmake -DTRITON_ENABLE_GPU=ON -DTRITON_BACKEND_REPO_TAG=<GIT_BRANCH_NAME> -DTRITON_COMMON_REPO_TAG=<GIT_BRANCH_NAME> -DTRITON_CORE_REPO_TAG=<GIT_BRANCH_NAME> -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install .. make install ``` The following required Triton repositories will be pulled and used in the build. If the CMake variables below are not specified, "main" branch of those repositories will be used. \<GIT\_BRANCH\_NAME\> should be the same as the Python backend repository branch that you are trying to compile. * triton-inference-server/backend: `-DTRITON_BACKEND_REPO_TAG=<GIT_BRANCH_NAME>` * triton-inference-server/common: `-DTRITON_COMMON_REPO_TAG=<GIT_BRANCH_NAME>` * triton-inference-server/core: `-DTRITON_CORE_REPO_TAG=<GIT_BRANCH_NAME>` Set `-DCMAKE_INSTALL_PREFIX` to the location where the Triton Server is installed. In the released containers, this location is `/opt/tritonserver`. 3. Copy example model and configuration ``` mkdir -p models/add_sub/1/ cp examples/add_sub/model.py models/add_sub/1/model.py cp examples/add_sub/config.pbtxt models/add_sub/config.pbtxt ``` 4. Start the Triton Server ``` /opt/tritonserver/bin/tritonserver --model-repository=`pwd`/models ``` 5. Use the client app to perform inference ``` python3 examples/add_sub/client.py ``` ## Usage In order to use the Python backend, you need to create a Python file that has a structure similar to below: ```python import triton_python_backend_utils as pb_utils class TritonPythonModel: """Your Python model must use the same class name. Every Python model that is created must have "TritonPythonModel" as the class name. """ @staticmethod def auto_complete_config(auto_complete_model_config): """`auto_complete_config` is called only once when loading the model assuming the server was not started with `--disable-auto-complete-config`. Implementing this function is optional. No implementation of `auto_complete_config` will do nothing. This function can be used to set `max_batch_size`, `input` and `output` properties of the model using `set_max_batch_size`, `add_input`, and `add_output`. These properties will allow Triton to load the model with minimal model configuration in absence of a configuration file.