rl-plotter - Effortless Visualization of Reinforcement Learning Curves

Introduction to rl-plotter

rl-plotter is a practical and straightforward tool designed for easily plotting learning curves, specifically aimed at projects involving reinforcement learning (RL). The tool is beneficial for researchers and developers who are working on RL projects and require efficient ways to visualize their training processes through learning curves. It's connected with popular RL platforms like OpenAI's spinningup and baseline.

Installation

One can set up rl-plotter in two main ways:

Using PIP: Simply open the terminal and run the command:
```
pip install rl_plotter
```
From the Source: For developers who prefer installing directly from the source, they can use:
```
python setup.py install
```

How to Use rl-plotter

1. Add a Basic Logger

To begin using rl-plotter, one should integrate a logger within their RL evaluation code. This helps track and update learning progress:

Import the logger:
```
from rl_plotter.logger import Logger
```

Initialize the logger with your experiment details:

logger = Logger(exp_name="your_exp_name", env_name, seed, locals())

Constantly update the logger with the evaluation metrics during training:

logger.update(score=evaluation_score_list, total_steps=current_training_steps)

For those already using OpenAI's platforms, rl-plotter supports logging through the OpenAI-spinningup and OpenAI-baseline libraries.

2. Track Additional Variables (Optional)

If there's a need to monitor variables aside from the basic tracking, users can create a custom logger:

custom_logger=logger.new_custom_logger(filename, fieldnames=["variable 1", "variable 2", ..., "variable n"])
custom_logger.update(fieldvalues=variable_value_list, total_steps=current_training_steps)

3. Plot the Results

Once training is underway or completed, plotting the learning curves is streamlined:

Navigate to the directory containing your logs.
Execute the command to generate plots:
```
rl_plotter --save --show
```

Alternatively, plots similar in style to OpenAI-spinningup can be generated using:

rl_plotter_spinup --save --show

Example Commands

Basic Commands: These commands help create visualizations for specific environments like HalfCheetah or Ant:

rl_plotter --save --show --filter HalfCheetah
rl_plotter --save --show --filter Ant --avg_group --shaded_std

Practical Example: A more comprehensive example to plot graphs with shaded errors and standard deviations:

rl_plotter --show --save --avg_group --shaded_err --shaded_std

Custom Usage: Use the help command to explore and tailor the plotting settings to fit specific needs:

rl_plotter --help

This command provides a comprehensive guide to optional arguments that allow customization of graph dimensions, styles, keys, and more.

Features of rl-plotter

Customizable loggers and plot styles
Directory filtering for structured data analysis
Support for multi-experiment plots and different plotting kernels
Seamless integration with OpenAI's baseline and spinningup tools
Options for multiple y-keys and dual y-axis legends

Acknowledgment

The backbone of rl-plotter derives inspiration from pioneering works in the field like OpenAI's baselines and spinningup plotting utilities.

In conclusion, rl-plotter simplifies the visualization of reinforcement learning processes, providing users with flexibility and compatibility across various RL environments. Whether you are just starting to develop RL models or are involved in high-level research, rl-plotter can enhance your workflow with its powerful plotting capabilities.