Project Overview: Tract
Tract is an advanced Neural Network inference toolkit developed by Sonos. It is designed to efficiently read, optimize, and run neural network models, primarily in ONNX and NNEF formats. Initially known as tfdeploy or Tensorflow-deploy-rust, Tract has since evolved into a powerful tool for machine learning applications.
Key Features
Model Compatibility and Performance
Tract excels in processing ONNX models, successfully passing approximately 85% of all ONNX backend tests. It fully supports popular models like AlexNet, DenseNet121, ResNet50, and more. However, it does have certain limitations with Tensor Sequences and Optional Tensors due to its structural inclination towards purely tensor-based operations. Despite these gaps, Tract thrives on contributions and collaborations to enhance its capabilities further.
For NNEF, Tract goes beyond by enabling support for most of the standard specifications. It introduces Tract-OPL, an extension that helps cover the full spectrum of neural network support. This means that any network understandable by Tract can eventually be serialized to Tract-OPL, aiming to bridge the gap between training and inference systems.
Support for Various Frameworks
Tract is versatile, allowing it to read models from various neural network frameworks, such as TensorFlow and Keras, and convert them to ONNX or NNEF/OPL for streamlined processing. While it must fully support TensorFlow 1.x models, it efficiently runs applications like Google Inception v3 and Snips wake word models, showcasing its adaptability.
Examples and Use Cases
Several example implementations using well-known models illustrate Tract's capabilities. These include MobileNet v2 with both ONNX and TensorFlow, BERT models via ONNX, and ResNet through PyTorch. Tract is highly beneficial in practical applications, particularly in environments constrained by computational resources, such as:
- Keyword Spotting on Microcontrollers: Demonstrating Tract's use in ARM Cortex-M Microcontrollers, allowing models to be deployed on simple devices like the Raspberry Pi with impressively short inference times.
- Snips Wake Word Models: Utilized by Sonos for efficient wake word detection, Tract's capabilities enable powerful processing on devices like the Raspberry Pi Zero.
Performance Metrics
Tract compares favorably against TensorFlow Lite in performance benchmarks on various devices. For instance, Inception v3 loaded in Tract runs faster on a Raspberry Pi Zero when compared to TensorFlow Lite, highlighting its optimized execution and resource efficiency.
Licensing and Contribution
Tract operates under a dual-license model, allowing users to choose between the Apache 2.0 and MIT licenses. This encourages open collaboration and contribution to the project. Any contributions made will be dual-licensed under these terms, fostering a community-driven development environment.
Conclusion
Tract represents a formidable toolkit for neural network inference, simplifying the transition from model design and training to real-world application. Its ongoing development and commitment to efficiency make it a valuable asset for anyone working in machine learning, from hobbyists to industrial developers.