Confection: The Sweet Configuration System for Python
Confection is a lightweight library designed to simplify the creation and management of configuration settings in Python, especially for complex applications like machine learning. Configuration management can be a daunting task, particularly when dealing with numerous hyperparameters and settings spread across various system levels. Confection seeks to streamline this process by providing a flexible framework to handle configuration trees effortlessly.
The Need for a Configuration System
For developers, especially those working with machine-learning models, the ability to adjust settings such as hyperparameters is crucial. These parameters might be buried deep within the codebase and need to be accessible through the command line interface (CLI) or a REST API. This complexity often results in a rigid system that's hard to adapt or update without breaking existing functionalities. Confection addresses these challenges by offering a configuration system that is both flexible and robust, allowing settings to be easily amended without disrupting backward compatibility.
Key Features and Workflow
Confection excels by allowing developers to articulate arbitrary trees of objects simply. Using decorator syntax, functions can be registered and versioned, offering a way to enhance and update configurations while retaining backward compatibility. This system is most akin to Google’s Gin, though Confection emphasizes a streamlined workflow with a subset of Gin's functionalities.
Installation and Setup
Getting started with Confection is straightforward. You can install it via pip or conda:
pip install confection
conda install -c conda-forge confection
Usage Example
Confection uses a .cfg file format to define configurations, similar to INI files. Here's a simple example:
[training]
patience = 10
dropout = 0.2
use_vectors = false
[training.logging]
level = "INFO"
[nlp]
use_vectors = ${training.use_vectors}
lang = "en"
This file describes a configuration where, for instance, the use_vectors setting in the nlp section references the same setting in the training section. The result is a structured dictionary that reflects the config, allowing easy access and modification within your Python scripts.
Config System Highlights
The Confection configuration format differs from the traditional Python configparser in several ways:
- JSON-Formatted Values: Confection uses JSON to interpret values, supporting complex data types like lists and maps.
- Structured Sections: It uses dot notation for nested sections, offering a clear hierarchy.
- Function References: Keys starting with
@reference registered functions within a registry, adding dynamic capabilities to configurations.
Real-World Application
For instance, imagine you want to specify a custom optimizer in your machine-learning model. You could define it in config.cfg:
[optimizer]
@optimizers = "my_cool_optimizer.v1"
learn_rate = 0.001
gamma = 1e-8
You could then load this configuration and use it in your Python code:
import dataclasses
import catalogue
from confection import registry, Config
# Define an optimizer and register it
registry.optimizers = catalogue.create("confection", "optimizers", entry_points=False)
@dataclasses.dataclass
class MyCoolOptimizer:
learn_rate: float
gamma: float
@registry.optimizers.register("my_cool_optimizer.v1")
def make_my_optimizer(learn_rate: float, gamma: float):
return MyCoolOptimizer(learn_rate, gamma)
# Load and resolve the config
config = Config().from_disk("./config.cfg")
resolved = registry.resolve(config)
optimizer = resolved["optimizer"]
Under the hood, Confection fetches the registered function and passes the defined arguments to it, ensuring type safety and correctness.
Additional Resources
For more advanced use cases, Confection integrates smoothly with tools like Pydantic for validation and offers features for handling recursive blocks, custom registry functions, and complex type annotations. The documentation provides extensive guidelines on these topics, making Confection a versatile tool for developers in need of a flexible configuration system.
Confection by Explosion is a powerful tool that enables developers to manage configurations in an intuitive, scalable way. Whether dealing with complex machine-learning configurations or simple script settings, Confection simplifies the process, allowing developers to focus on building and refining their projects.