人工智能+深度学习+NVIDIA Tensor内核深度学习示例+源码

# Deploying the TFT model on Triton Inference Server This folder contains instructions for deployment to run inference on Triton Inference Server as well as a detailed performance analysis. The purpose of this document is to help you with achieving the best inference performance. ## Table of contents - [Solution overview](#solution-overview) - [Introduction](#introduction) - [Deployment process](#deployment-process) - [Setup](#setup) - [Quick Start Guide](#quick-start-guide) - [Performance](#performance) - [Offline scenario](#offline-scenario) - [Offline: NVIDIA A30, NVIDIA TensorRT with FP16, Dataset: electricity](#offline-nvidia-a30-nvidia-tensorrt-with-fp16-dataset-electricity) - [Offline: NVIDIA A30, NVIDIA TensorRT with FP16, Dataset: traffic](#offline-nvidia-a30-nvidia-tensorrt-with-fp16-dataset-traffic) - [Offline: NVIDIA A30, PyTorch with FP16, Dataset: electricity](#offline-nvidia-a30-pytorch-with-fp16-dataset-electricity) - [Offline: NVIDIA A30, PyTorch with FP16, Dataset: traffic](#offline-nvidia-a30-pytorch-with-fp16-dataset-traffic) - [Offline: NVIDIA DGX-1 (1x V100 32GB), NVIDIA TensorRT with FP16, Dataset: electricity](#offline-nvidia-dgx-1-1x-v100-32gb-nvidia-tensorrt-with-fp16-dataset-electricity) - [Offline: NVIDIA DGX-1 (1x V100 32GB), NVIDIA TensorRT with FP16, Dataset: traffic](#offline-nvidia-dgx-1-1x-v100-32gb-nvidia-tensorrt-with-fp16-dataset-traffic) - [Offline: NVIDIA DGX-1 (1x V100 32GB), PyTorch with FP16, Dataset: electricity](#offline-nvidia-dgx-1-1x-v100-32gb-pytorch-with-fp16-dataset-electricity) - [Offline: NVIDIA DGX-1 (1x V100 32GB), PyTorch with FP16, Dataset: traffic](#offline-nvidia-dgx-1-1x-v100-32gb-pytorch-with-fp16-dataset-traffic) - [Offline: NVIDIA DGX A100 (1x A100 80GB), NVIDIA TensorRT with FP16, Dataset: electricity](#offline-nvidia-dgx-a100-1x-a100-80gb-nvidia-tensorrt-with-fp16-dataset-electricity) - [Offline: NVIDIA DGX A100 (1x A100 80GB), NVIDIA TensorRT with FP16, Dataset: traffic](#offline-nvidia-dgx-a100-1x-a100-80gb-nvidia-tensorrt-with-fp16-dataset-traffic) - [Offline: NVIDIA DGX A100 (1x A100 80GB), PyTorch with FP16, Dataset: electricity](#offline-nvidia-dgx-a100-1x-a100-80gb-pytorch-with-fp16-dataset-electricity) - [Offline: NVIDIA DGX A100 (1x A100 80GB), PyTorch with FP16, Dataset: traffic](#offline-nvidia-dgx-a100-1x-a100-80gb-pytorch-with-fp16-dataset-traffic) - [Offline: NVIDIA T4, NVIDIA TensorRT with FP16, Dataset: electricity](#offline-nvidia-t4-nvidia-tensorrt-with-fp16-dataset-electricity) - [Offline: NVIDIA T4, NVIDIA TensorRT with FP16, Dataset: traffic](#offline-nvidia-t4-nvidia-tensorrt-with-fp16-dataset-traffic) - [Offline: NVIDIA T4, PyTorch with FP16, Dataset: electricity](#offline-nvidia-t4-pytorch-with-fp16-dataset-electricity) - [Offline: NVIDIA T4, PyTorch with FP16, Dataset: traffic](#offline-nvidia-t4-pytorch-with-fp16-dataset-traffic) - [Online scenario](#online-scenario) - [Online: NVIDIA A30, NVIDIA TensorRT with FP16, Dataset: electricity](#online-nvidia-a30-nvidia-tensorrt-with-fp16-dataset-electricity) - [Online: NVIDIA A30, NVIDIA TensorRT with FP16, Dataset: traffic](#online-nvidia-a30-nvidia-tensorrt-with-fp16-dataset-traffic) - [Online: NVIDIA A30, PyTorch with FP16, Dataset: electricity](#online-nvidia-a30-pytorch-with-fp16-dataset-electricity) - [Online: NVIDIA A30, PyTorch with FP16, Dataset: traffic](#online-nvidia-a30-pytorch-with-fp16-dataset-traffic) - [Online: NVIDIA DGX-1 (1x V100 32GB), NVIDIA TensorRT with FP16, Dataset: electricity](#online-nvidia-dgx-1-1x-v100-32gb-nvidia-tensorrt-with-fp16-dataset-electricity) - [Online: NVIDIA DGX-1 (1x V100 32GB), NVIDIA TensorRT with FP16, Dataset: traffic](#online-nvidia-dgx-1-1x-v100-32gb-nvidia-tensorrt-with-fp16-dataset-traffic) - [Online: NVIDIA DGX-1 (1x V100 32GB), PyTorch with FP16, Dataset: electricity](#online-nvidia-dgx-1-1x-v100-32gb-pytorch-with-fp16-dataset-electricity) - [Online: NVIDIA DGX-1 (1x V100 32GB), PyTorch with FP16, Dataset: traffic](#online-nvidia-dgx-1-1x-v100-32gb-pytorch-with-fp16-dataset-traffic) - [Online: NVIDIA DGX A100 (1x A100 80GB), NVIDIA TensorRT with FP16, Dataset: electricity](#online-nvidia-dgx-a100-1x-a100-80gb-nvidia-tensorrt-with-fp16-dataset-electricity) - [Online: NVIDIA DGX A100 (1x A100 80GB), NVIDIA TensorRT with FP16, Dataset: traffic](#online-nvidia-dgx-a100-1x-a100-80gb-nvidia-tensorrt-with-fp16-dataset-traffic) - [Online: NVIDIA DGX A100 (1x A100 80GB), PyTorch with FP16, Dataset: electricity](#online-nvidia-dgx-a100-1x-a100-80gb-pytorch-with-fp16-dataset-electricity) - [Online: NVIDIA DGX A100 (1x A100 80GB), PyTorch with FP16, Dataset: traffic](#online-nvidia-dgx-a100-1x-a100-80gb-pytorch-with-fp16-dataset-traffic) - [Online: NVIDIA T4, NVIDIA TensorRT with FP16, Dataset: electricity](#online-nvidia-t4-nvidia-tensorrt-with-fp16-dataset-electricity) - [Online: NVIDIA T4, NVIDIA TensorRT with FP16, Dataset: traffic](#online-nvidia-t4-nvidia-tensorrt-with-fp16-dataset-traffic) - [Online: NVIDIA T4, PyTorch with FP16, Dataset: electricity](#online-nvidia-t4-pytorch-with-fp16-dataset-electricity) - [Online: NVIDIA T4, PyTorch with FP16, Dataset: traffic](#online-nvidia-t4-pytorch-with-fp16-dataset-traffic) - [Advanced](#advanced) - [Step by step deployment process](#step-by-step-deployment-process) - [Latency explanation](#latency-explanation) - [Release notes](#release-notes) - [Changelog](#changelog) - [Known issues](#known-issues) ## Solution overview ### Introduction The [NVIDIA Triton Inference Server](https://github.com/NVIDIA/triton-inference-server) provides a datacenter and cloud inferencing solution optimized for NVIDIA GPUs. The server provides an inference service via an HTTP or gRPC endpoint, allowing remote clients to request inferencing for any number of GPU or CPU models being managed by the server. This README provides step-by-step deployment instructions for models generated during training (as described in the [model README](../readme.md)). Additionally, this README provides the corresponding deployment scripts that ensure optimal GPU utilization during inferencing on Triton Inference Server. ### Deployment process The deployment process consists of two steps: 1. Conversion. The purpose of conversion is to find the best performing model format supported by Triton Inference Server. Triton Inference Server uses a number of runtime backends such as [TensorRT](https://developer.nvidia.com/tensorrt), [LibTorch](https://github.com/triton-inference-server/pytorch_backend) and [ONNX Runtime](https://github.com/triton-inference-server/onnxruntime_backend) to support various model types. Refer to the [Triton documentation](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton) for a list of available backends. 2. Configuration. Model configuration on Triton Inference Server, which generates necessary [configuration files](https://github.com/triton-inference-server/server/blob/master/docs/model_configuration.md). After deployment Triton inference server is used for evaluation of converted model in two steps: 1. Accuracy tests. Produce results which are tested against given accuracy thresholds. 2. Performance tests. Produce latency and throughput results for offline (static batching) and online (dynamic batching) scenarios. All steps are executed by provided runner script. Refer to [Quick Start Guide](#quick-start-guide) ## Setup Ensure you have the following components: * [NVIDIA Docker](https://github.com/NVIDIA/nvidia-docker) * [PyTorch NGC container 21.12](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) * [Triton In