genrl - Efficient and Modular PyTorch Reinforcement Learning Library for Reliable Algorithm Implementation and Reproduction

Introduction to GenRL

GenRL is an innovative reinforcement learning library developed with PyTorch. It's designed to streamline the process of reproducing and generalizing algorithm implementations, aiming to make reinforcement learning (RL) more accessible to researchers and developers alike. Currently in version 0.0.2, GenRL is a dynamic project expected to undergo various updates, including breaking changes.

Key Features of GenRL

GenRL offers several features that cater to both beginners and seasoned RL practitioners:

PyTorch-First Approach: The library is modular and extensible, allowing users to customize implementations in Python easily.
Comprehensive Tutorials and Examples: GenRL includes over 20 tutorials that cover the basics of RL to more advanced state-of-the-art deep RL algorithms. Each tutorial is paired with detailed explanations to facilitate learning.
Unified Trainer and Logging Class: GenRL promotes code reusability and offers a high-level user interface, making it easier to implement RL algorithms.
Ready-made Algorithm Implementations: The library provides pre-implemented popular RL algorithms, helping expedite the experiment process.
Accelerated Benchmarking: Integrated features like automated hyperparameter tuning and environment setups enhance benchmarking speed.
GenRL aspires to accommodate new algorithm implementations in under 100 lines of code.

How to Contribute

GenRL welcomes contributions from the community for code, documentation, and testing. Interested contributors can start by reviewing existing issues and opening pull requests. For guidance, the Contributing Guidelines are available for reference.

Installation

GenRL is compatible with Python 3.6 or later, and relies on PyTorch as well as OpenAI Gym. To streamline installation, pip can be used, following these commands:

$ pip install genrl

GenRL is frequently updated. To upgrade to the latest version, run:

$ pip install -U genrl

For those interested in experimenting with the latest unreleased version, GenRL is available for installation from source:

$ git clone https://github.com/SforAiDl/genrl.git
$ cd genrl
$ python setup.py install

Usage Examples

GenRL supports various models and environments. For instance, to train a Soft Actor-Critic (SAC) model on the Pendulum-v0 environment and log rewards using TensorBoard:

import gym
from genrl.agents import SAC
from genrl.trainers import OffPolicyTrainer
from genrl.environments import VectorEnv

env = VectorEnv("Pendulum-v0")
agent = SAC('mlp', env)
trainer = OffPolicyTrainer(agent, env, log_mode=['stdout', 'tensorboard'])
trainer.train()

Similarly, to train a Tabular Dyna-Q model on FrozenLake-v0:

import gym
from genrl.agents import QLearning
from genrl.trainers import ClassicalTrainer

env = gym.make("FrozenLake-v0")
agent = QLearning(env)
trainer = ClassicalTrainer(agent, env, mode="dyna", model="tabular", n_episodes=10000)
episode_rewards = trainer.train()
trainer.plot(episode_rewards)

Tutorials and Learning Resources

GenRL provides an array of tutorials spanning various topics like:

Multi-Armed Bandits, exploring techniques like Upper Confidence Bound and Thompson Sampling.
Contextual Bandits, covering Linear Posterior Inference and Variational Inference.
Deep Reinforcement Learning, including techniques like Vanilla Policy Gradients and Proximal Policy Optimization.

Supported Algorithms

Deep RL Algorithms

Various versions of DQN (Deep Q Networks)
VPG (Vanilla Policy Gradients)
A2C (Advantage Actor-Critic)
PPO (Proximal Policy Optimization)
DDPG (Deep Deterministic Policy Gradients)
TD3 (Twin Delayed DDPG)
SAC (Soft Actor Critic)

Classical RL Algorithms

SARSA
Q Learning

Bandit RL Algorithms

Multi-Armed Bandits (e.g., Epsilon Greedy, UCB, Thompson Sampling)
Contextual Bandits

Deep Contextual Bandits

Techniques like Variation Inference and noise sampling for neural networks

Related Libraries and Credits

GenRL is inspired by various notable projects:

By incorporating these features and learning resources, GenRL stands as a powerful tool for anyone interested in advancing their understanding and application of reinforcement learning.