foolbox - Python Library for Adversarial Attack Testing on PyTorch, TensorFlow, and JAX Models

Foolbox: Benchmarking Machine Learning Model Robustness

Overview

Foolbox is a highly efficient Python library designed for running adversarial attacks against machine learning models, particularly deep neural networks. Built on top of EagerPy, Foolbox seamlessly operates with models developed in PyTorch, TensorFlow, and JAX. It aims to evaluate and enhance the robustness of these models by simulating adversarial threats.

Design and Features

Native Performance: Foolbox 3 has undergone a complete rewrite to ensure native performance with EagerPy. This integration allows the library to function efficiently across PyTorch, TensorFlow, and JAX without duplicating code. It supports real batch processing, making it highly effective for large-scale operations.
State-of-the-Art Attacks: The library offers a comprehensive array of advanced gradient-based and decision-based adversarial attacks, allowing users to test various vulnerability aspects of machine learning models.
Type Checking: With extensive type annotations, Foolbox helps identify potential coding errors before execution, ensuring smoother and more reliable operations.

Getting Started

For beginners, Foolbox offers resources to ease the learning curve:

Guide and Documentation: Beginners can get started with Foolbox using the official guide available online. Detailed API documentation is also accessible on ReadTheDocs for those who seek in-depth technical information.
Tutorials: There is a well-crafted Jupyter notebook tutorial available on GitHub and accessible via Google Colab. This is an excellent resource for hands-on experience.

Installation

Installing Foolbox is straightforward. It supports Python versions 3.8 and newer, even though it might work on versions as old as 3.6. To begin, users only need to run the pip install command:

pip install foolbox

It's important to note that PyTorch, TensorFlow, or JAX needs to be installed separately based on one's requirement, as these frameworks are not dependencies for Foolbox.

Example Usage

A typical usage of Foolbox might look like this:

import foolbox as fb

model = ...
fmodel = fb.PyTorchModel(model, bounds=(0, 1))

attack = fb.attacks.LinfPGD()
epsilons = [0.0, 0.001, 0.01, 0.03, 0.1, 0.3, 0.5, 1.0]
_, advs, success = attack(fmodel, images, labels, epsilons=epsilons)

More examples, like the ResNet-18 example, can be found in the examples folder online.

Contributions

Foolbox welcomes contributions from developers and researchers. Whether you want to introduce new adversarial attacks or improve existing functionalities, the development community is eager for collaboration. Guidelines for contributing can be found in their development section online.

Community and Support

For questions and community support, users are encouraged to open issues on GitHub. There is an ongoing plan to transition to GitHub Discussions for more streamlined communication as it becomes available.

Performance and Compatibility

Foolbox 3.0 provides significantly improved performance over its predecessors, making it a preferable choice for performance-critical applications. It is compatible with recent versions of popular machine learning frameworks, ensuring it stays current with industry standards.

Overall, Foolbox is an invaluable tool for evaluating the robustness of machine learning models, contributing significantly to the field of adversarial machine learning.