uncertainty-baselines

Uncertainty Baselines: A Friendly Exploration

Introduction

Uncertainty Baselines is a project designed to assist researchers in the field of uncertainty and robustness. It provides a solid foundation for developing new ideas, applications, and collaboration in this domain. The project aims to address three key areas:

1. Offering high-quality implementations of both standard and cutting-edge methods on typical tasks.
2. Minimizing dependencies between files to ensure baselines are easily customizable without needing other parts of the codebase.
3. Recommending best practices for uncertainty and robustness assessment.

The inspiration behind Uncertainty Baselines is the existing disarray in uncertainty and robustness research. Many implementations on platforms like GitHub are made for specific studies and lack code accessibility. This fragmented landscape makes it challenging for researchers to compare their work or build upon others without reinventing the wheel.

Installation

To install the latest development version of Uncertainty Baselines, use the command below. Please note, it doesn’t come with a stable version or official release, and APIs may change over time.

pip install "git+https://github.com/google/uncertainty-baselines.git#egg=uncertainty_baselines"

Usage

Baselines

The baselines are stored in a specific directory and are categorized by the training dataset. These baselines include models and scripts, like the Wide ResNet model achieving 96.0% accuracy on the CIFAR-10 dataset.

Launching with TPUs is often necessary for reproducing baselines, accessible via:

1. Colab: Free TPUs are available on Colab, ideal for small-scale experiments.
2. Google Cloud: Offers flexibility and scalability with cloud TPU setup.
3. Flag Adjustment: Allows switching between TPU cores and GPUs as needed.

Datasets

The ub.datasets module houses datasets compatible with the TensorFlow Datasets API. They come with basic logic for preprocessing and can be used with other frameworks like Jax and PyTorch.

import uncertainty_baselines as ub
dataset_builder = ub.datasets.Cifar10Dataset(split='train', validation_percent=0.1)
train_dataset = dataset_builder.load(batch_size=FLAGS.batch_size)

Models

Models under the ub.models module align with the tf.keras.Model API, providing a standard interface for building diverse deep learning models.

import uncertainty_baselines as ub
model = ub.models.wide_resnet(input_shape=(32, 32, 3), depth=28, width_multiplier=10, num_classes=10, l2=1e-4)

Metrics

Uncertainty Baselines define several metrics to assess model performance thoroughly:

1. Number of Parameters: Counts the model parameters.
2. Test Accuracy: Measures the accuracy over the test set.
3. Expected Calibration Error (ECE): Examines the model's confidence against accuracy.
4. Negative Log-Likelihood (NLL): Evaluates the likelihood loss.
5. Runtime: Details the total time for training and testing.

Metrics can be visualized using TensorBoard, allowing deeper insights into model predictions and accuracy.

Contributing

Contributors interested in adding to the project need to ensure code formatting with yapf, add new baselines or datasets by following simple guidelines, and ensure tests are included to validate new models and data integrations.

Conclusion

Uncertainty Baselines offers a structured, accessible, and easy-to-utilize resource for researchers in the uncertainty and robustness fields. By standardizing these baselines, it helps bridge gaps in experiment reproducibility and facilitates advancement in machine learning research focused on uncertainty.