#NVIDIA GPUs

Triton Inference Server is an open-source inference software that facilitates deploying AI models across various platforms. It supports multiple deep learning and machine learning frameworks, including TensorFlow and PyTorch, ensuring seamless integration and optimized performance on NVIDIA GPUs, ARM CPUs, or AWS Inferentia. Designed for real-time, batched, and streaming queries, the server supports features such as dynamic batching, custom backends, sequence batching, and ensemble models. It also provides comprehensive metrics for performance insights and offers both HTTP/REST and GRPC protocols. As part of NVIDIA AI Enterprise, Triton contributes to enhancing data science workflows and AI deployment.

This study evaluates the inference performance of different GPUs, such as NVIDIA and Apple Silicon, for LLaMA 3 models using llama.cpp. It includes detailed benchmarks on RunPod and various MacBook models. The focus is on average speeds for 1024 token generation and prompt evaluation, presented in tokens per second. The insights can assist in optimizing GPU choices for improved large language model operations.

Achieve rapid audio transcription with Whisper and Flash Attention on supported devices. Utilizing fp16 and batching optimizations, transcribe 150 minutes of audio significantly faster. The tool allows for automatic speech recognition with options for language detection and speaker diarization. Compatible with CUDA and mps, it streamlines installation and execution from any directory.

Stable-fast provides top-tier inference capabilities for diffuser models, such as the StableVideoDiffusionPipeline, with compilation in seconds, unlike TensorRT. It natively supports dynamic shapes, LoRA, and ControlNet. Primed for HuggingFace Diffusers on NVIDIA GPUs, this framework leverages techniques like CUDNN Convolution Fusion and low precision Fused GEMM for enhancements. Designed for compatibility with multiple PyTorch editions and acceleration tools, Stable-fast requires minimal adjustments for maximum performance.

Provides efficient inference for BERT-like models with Python and C++ APIs using advanced architectural optimizations. Compatible with both fixed and variable-length transformers, the library surpasses other frameworks, as highlighted in IEEE IPDPS 2023. Implemented at ByteDance, it exceeds PyTorch and TensorFlow performance on NVIDIA GPUs. The setup is straightforward, requiring CUDA 11.6, CMake 3.13+, and PyTorch 1.8+.

GauHuman utilizes monocular videos for effective Gaussian splatting with training times of 1-2 minutes and real-time rendering at up to 189 FPS. It offers detailed guidance on environment setup, training, and evaluation using a range of datasets including ZJU-Mocap-Refine and MonoCap. Compatibility with NVIDIA GPUs is mandatory, with comprehensive setup instructions provided. GauHuman excels in efficient rendering of human movements and integrates foundational technologies such as Gaussian-Splatting and HumanNeRF.