PPO for Beginners
Introduction
Eric Yu has put together a helpful repository aimed at beginners eager to learn and implement Proximal Policy Optimization (PPO) using PyTorch. His goal with this project is to offer a straightforward, well-documented PPO code, aimed particularly at those who find existing complex implementations overwhelming. The repository acts as a foundation for understanding the basic workings of PPO, allowing learners to implement it practically without any complicated extras.
For those interested in following this journey in more detail, Eric suggests starting with his comprehensive series on Medium. He assumes that users have a basic understanding of Python and Reinforcement Learning (RL), which includes familiar concepts of policy gradient (pg) algorithms and PPO on a theoretical level.
Eric's implementation presumes the use of continuous observation and action spaces, though modifications can be made for discrete alternatives. His work closely follows the pseudocode from OpenAI’s Spinning Up guide for PPO.
Usage
Eric suggests a couple of steps to begin using the repository. First, create a Python virtual environment to manage dependencies smoothly:
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
To train a model from scratch, simply execute:
python main.py
For testing a model using an existing actor:
python main.py --mode test --actor_model ppo_actor.pth
If you have existing actor and critic models ready, use:
python main.py --actor_model ppo_actor.pth --critic_model ppo_critic.pth
Keep in mind, changing hyperparameters or environments should be done within main.py since command line arguments are not used for customization to keep commands concise.
How it Works
The central script, main.py, is the starting point. It handles parsing of arguments via arguments.py and sets up the environment and PPO model. Depending on the specified mode, it either trains or tests the model using a function called learn. This approach echoes the training method used in PPO2 from stable_baselines.
arguments.py aids in command line argument parsing, while ppo.py houses the core PPO functionalities. Eric encourages readers to explore his Medium series or utilize Python’s debugger pdb to gain deeper insights into how the code operates.
Additionally, network.py presents a sample feed-forward neural network crafted for actor and critic networks in PPO. The policy evaluation process happens separately within eval_policy.py.
The graph_code directory contains the code for data collection and graph generation, which complements the detailed analyses shared in Eric’s Medium article.
Environments
Eric provides a list of environments that can be experimented with, noting the requirement of Box for both observation and action spaces within his PPO implementation. For those interested in adjusting hyperparameters, they are available here.
Results
The results and detailed explorations of Eric's work are documented in his Medium article, offering further insights into the project’s efficacy.
Contact
For questions or further engagement, Eric Yu can be reached via email at [email protected] or through his LinkedIn profile.