Edgeai TIDL工具和示例 - 此存储库包含为深度学习运行时开发的工具和示例_python_代码_下载

# Python Examples - [Python Examples](#python-examples) - [Introduction](#introduction) - [OSRT based user work flow](#osrt-based-user-work-flow) - [Model Compilation on PC](#model-compilation-on-pc) - [Model Inference on EVM](#model-inference-on-evm) - [User options for TFLite and ONNX Runtime](#user-options-for-tflite-and-onnx-runtime) - [Required](#required) - [Optional](#optional) - [User options for TVM](#user-options-for-tvm) - [Trouble Shooting](#trouble-shooting) ## Introduction TIDL provides multiple deployment options with industry defined inference engines as listed below. These inference engines are being referred as Open Source Run Times in this document. * **TFLite Runtime**: [TensorFlow Lite](https://www.tensorflow.org/lite/guide/inference) based inference with heterogeneous execution on cortex-A** + C7x-MMA, using TFlite Delegates [TFLite Delgate](https://www.tensorflow.org/lite/performance/delegates) API * **ONNX RunTime**: [ONNX Runtime]( https://www.onnxruntime.ai/) based inference with heterogeneous execution on cortex-A** + C7x-MMA. * **TVM/Neo-AI RunTime**: [TVM]( https://tvm.apache.org)/[Neo-AI-DLR]( https://github.com/neo-ai/neo-ai-dlr) based inference with heterogeneous execution on cortex-A** + C7x-MMA >** *TDA4VM has cortex-A72 as its MPU, refer to the device TRM to know which cortex-A MPU* it contains. These heterogeneous execution enables: 1. OSRT as the top level inference API for user applications 2. Offloading subgraphs to C7x/MMA for accelerated execution with TIDL 3. Runs optimized code on ARM core for layers that are not supported by TIDL ## OSRT based user work flow The diagram below illustrates the TFLite based work flow as an example. ONNX RunTime and TVM/Neo-AI RunTime also follows similar work flow. The User needs to run the model compilation (sub-graph(s) creation and quantization) on PC and the generated artifacts can be used for inference on the device.  ## Model Compilation on PC  1. Prepare the Environment for the Model compilation by follwoing the setup section [here](../../README.md#setup) 2. Run for model compilation in the corresponding example folder – This step generates artifacts needed for inference in the \<repo base>\/model-artifacts folder. Each subgraph is identified in the artifacts using the tensor index of its output in the model ``` cd examples/osrt_python/tfl python3 tflrt_delegate.py -c ``` 3. Run Inference on PC - Optionally user can test the inference in host emulation mode and check the output; the output images will be saved in the corresponding specified artifacts folder ``` python3 tflrt_delegate.py ``` 4. Run Inference on PC without offload - Optionally user can test the inference in host emulation mode without using any delegation to TI Delegate ``` python3 tflrt_delegate.py -d ``` ## Model Inference on EVM The artifacts generated by python scripts in the above section can be inferred using either python or C/C++ APIs. The following steps are for running inference using python API.  1. Copy the below folders from PC to the EVM where this repo is cloned ``` ./model-artifacts ./models ``` 2. Run the inference script in the corresponding example folder on the EVM and check the results, performance etc. ``` cd examples/osrt_python/tfl python3 tflrt_delegate.py ``` Note : These scripts are only for basic functionally testing and performance check. Accuracy of the models can be benchmarked using the python module released here [edgeai-benchmark](https://github.com/TexasInstruments/edgeai-benchmark) ## User options for TFLite and ONNX Runtime An example call to TFLite interpreter from the python interface using delegate mechanism: interpreter = tflite.Interpreter(model_path='path_to_model', \ experimental_delegates=[tflite.load_delegate('libtidl_tfl_delegate.so.1.0', delegate_options)]) An example call to ONNX runtime session from the python interface : EP_list = ['TIDLExecutionProvider','CPUExecutionProvider'] sess = rt.InferenceSession('path_to_model' ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so) 'delegate_options' in the inference session call comprise of the following options (required and optional). All these options are common for TFLite and ONNX runtime. ### Required The following options need to be specified by user while creating TFLite interpreter: | Name | Value | |:-------------------|:--------------------------------------------------------| | tidl_tools_path | to be set to ${TIDL_TOOLS_PATH} - Path from where to pick TIDL related tools | | artifacts_folder | folder where user intends to store all the compilation artifacts | ### Optional The following options are set to default values, to be specified if modification needed by user. Below optional arguments are specific to model compilation and not applicable to inference except the 'debug_level' | Name | Description | Default values | |:-------------------|:--------------------------------------------------------|:--------------------------:| | platform | "J7" | "J7" | | version | TIDL version - open source runtimes supported from version 7.2 onwards | (7,3) | | tensor_bits | Number of bits for TIDL tensor and weights - 8/16 | 8 | | debug_level | 0 - no debug, 1 - rt debug prints, >=2 - increasing levels of debug and trace dump | 0 | | max_num_subgraphs | offload up to \<num\> tidl subgraphs | 16 | | deny_list | force disable offload of a particular operator to TIDL [^2] | "" - Empty list | | accuracy_level | 0 - basic calibration, 1 - higher accuracy(advanced bias calibration), 9 - user defined [^3] | 1 | | advanced_options:calibration_frames | Number of frames to be used for calibration - min 10 frames recommended | 20 | | advanced_options:calibration_iterations | Number of bias calibration iterations | 50 | | advanced_options:output_feature_16bit_names_list | List of names of the layers (comma separated string) as in the original model whose feature/activation output user wants to be in 16 bit [^4] | "" | | advanced_options:params_16bit_names_list | List of names of the output layers (separated by comma or space or tab) as in the original model whose parameters user wants to be in 16 bit [^1] | "" | | advanced_options:quantization_scale_type | 0 for non-power-of-2, 1 for power-of-2 | 0 | | advanced_options:high_resolution_optimization | 0 for disable, 1 for enable | 0 | | advanced_options:pre_batchnorm_fold | Fold batchnorm layer into following convolution layer, 0 for disable, 1 for enable | 1 | | advanced_options:add_data_convert_ops | Adds the Input and Output format conversions to Model and performs the same in DSP instead of ARM. This is currently a experimental feature. | 0 | | object_detection:confidence_threshold | Override "nms_score_threshold" parameter threshold in tflite detection post processing layer | Read from model | | object_detection:nms_threshold | Override "nms_iou_threshold" parameter threshold in tflite detection post processing layer |