fairscale
FairScale is a PyTorch extension that improves scalability and performance in large-scale model training. It features advanced scaling techniques with composable modules and easy-to-use APIs. Emphasizing usability, modularity, and efficiency, FairScale aids researchers in scaling models even with resource constraints. Its FullyShardedDataParallel (FSDP) element provides large neural network scalability and is now part of PyTorch for widespread use. Operating under a BSD-3-Clause License, FairScale welcomes ongoing contributions.
