#Attention Mechanism

Explore comprehensive implementations of Transformers in PyTorch, focusing on building them from scratch. The project features highly commented code for Bidirectional and Causal Attention layers and offers standalone implementations of models like GPT-2 and BERT, designed for seamless compilation. Perfect for those interested in the inner workings of attention mechanisms and transformer models.

Explore a training-free method to improve text-to-image diffusion models using T-GATE. It utilizes temporally gated attention to streamline the inference process into semantics-planning and fidelity-improving phases, enhancing model speed by 10-50% without compromising image quality. T-GATE integrates easily with existing frameworks and is compatible with CNN-based U-Net, Transformer, and Consistency Models. It now supports PixArt-Sigma and StableVideoDiffusion models, providing a valuable option for efficient image generation in various applications.

Inspectus is a tool designed for visualizing machine learning processes, integrating seamlessly into Jupyter notebooks with an easy Python API. It offers tools for visualizing attention mechanisms, analyzing token metrics, and plotting data distributions, aiding in comprehending deep learning models. Features such as heatmaps for query and key tokens and customizable attention maps improve model insight. Additionally, its ability to plot data distributions assists in outlier detection and model behavior analysis. Suitable for researchers and developers, Inspectus supports Huggingface models, providing a detailed view of deep learning frameworks.