tiny-tensorrt - Simple Deployment for ONNX Models with TensorRT Wrapper using C++ and Python

Introducing tiny-tensorrt

tiny-tensorrt is a user-friendly NVIDIA TensorRT wrapper designed for deploying ONNX models using C++ and Python APIs. Developed by zerollzeng, this project simplifies the process of model deployment, allowing users to perform complex tasks with just a few lines of code. While the project is no longer actively maintained, it offers a straightforward solution for those who need to implement TensorRT in their workflow.

Key Features

Ease of Use: tiny-tensorrt allows users to deploy models with minimal coding effort. The provided example shows how a few lines of C++ code can set up and execute a neural network.

Trt* net = new Trt();
net->SetFP16();
net->BuildEngine(onnxModel, engineFile);
net->CopyFromHostToDevice(input, inputBindIndex);
net->Forward();
net->CopyFromDeviceToHost(output, outputBindIndex);

Compatibility: Supports a variety of CUDA and TensorRT versions, making it flexible for use in different environments. Supported CUDA versions include 10.2, and up to 11.4, with TensorRT versions ranging from 7.0 to 8.4.
Language Support: Offers both C++ and Python APIs, catering to developers who prefer different programming languages.

Installation Guide

To use tiny-tensorrt, ensure that CUDA, CUDNN, and TensorRT are pre-installed. For a simplified setup, NVIDIA provides an official Docker image.

Follow these steps to install tiny-tensorrt:

Update your package manager and install necessary packages:

sudo apt-get update -y
sudo apt-get install cmake zlib1g-dev

For Python support, install Python and necessary packages:

sudo apt-get install python3 python3-pip
pip3 install numpy

Clone the project repository including its submodules:

git clone --recurse-submodules -j8 https://github.com/zerollzeng/tiny-tensorrt.git

Build the project:

cd tiny-tensorrt
mkdir build && cd build
cmake .. && make

Once installed, you can integrate tiny-tensorrt into your own project using libtinytrt.so and Trt.h files. For Python, use pytrt.so.

Documentation and Resources

For further details and in-depth understanding, users are encouraged to visit the Wiki of the project.

Licensing

While using tiny-tensorrt, users should adhere to the licensing requirements of third-party modules and TensorRT. The sections of the project authored by zerollzeng are open for use without restrictions.

Note: Although tiny-tensorrt offers a simple solution for deploying ONNX models, it is important to highlight that there are now more advanced alternatives available, such as TensorRT’s Python API and the trtexec/polygraphy tools for engine building. For the most updated issues or inquiries about TensorRT, users can file issues directly against NVIDIA/TensorRT.