deep-neuroevolution - Neuroevolution Methods in Deep Neural Network Training for Reinforcement Learning

Deep Neuroevolution Project Introduction

Overview

Deep Neuroevolution represents an innovative approach in the field of artificial intelligence, specifically focusing on training deep neural networks for reinforcement learning. This project implements two main algorithms: Evolution Strategies (ES) and Novelty-Seeking Evolution Strategies (NS-ES), both of which offer competitive alternatives to traditional deep learning methods. Originally developed by OpenAI, this project's code has been adapted to include variations such as the Deep Genetic Algorithm (DeepGA) and can run on local machines or through AWS.

Research Foundations

The backbone of this repository is constructed upon two pivotal research papers:

Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning - This paper explores the potential of genetic algorithms as an effective method for training neural networks, providing a robust alternative to the more commonly used gradient-based methods.
Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents - This study investigates enhanced exploration techniques in evolution strategies to better tackle complex reinforcement learning environments.

Key Features

Visual Inspector for NeuroEvolution (VINE)

VINE is an interactive visualization tool that offers a graphical representation of the neuroevolution process. Housed within the ./visual_inspector folder, this feature allows users to observe the evolutionary results visually, providing deeper insights into algorithm behavior and outcomes.

Accelerated Computing with GPU

For better performance, especially with computational-heavy tasks, this project includes a GPU-friendly implementation found in the ./gpu_implementation folder. This allows for faster data processing and more efficient algorithm execution.

Getting Started Locally

To start using this project on your local machine, follow these steps:

Clone the Repository: Obtain the project code using the command:

git clone https://github.com/uber-common/deep-neuroevolution.git

Set Up Environment: Create a virtual environment to manage the necessary dependencies:
```
python3 -m venv env
. env/bin/activate
```
Install Requirements: Use the following command to install all required packages:
```
pip install -r requirements.txt
```
Running Experiments: Execute various experiments using predefined scripts. Examples include Evolution Strategies for the Atari game Frostbite or Novelty-Seeking Evolution Strategies for a MuJoCo Humanoid environment.
Visualize Results: Employ python -m scripts.viz to graphically render the results of your experiments, which can be particularly insightful for understanding the performance of different strategies.

Running on Docker

The project also offers the flexibility of running within a Docker container. By leveraging Docker and Docker Compose, users can simplify environment setup and management. This involves:

Cloning the repository,
Starting the Docker container with docker-compose,
Executing experiments in a separate terminal session inside the Docker setup.

Conclusion

Deep Neuroevolution is a versatile, cutting-edge project for researchers and developers interested in alternatives to traditional neural network training methods. By incorporating genetic algorithms and evolution strategies, this project provides a powerful toolset for tackling a variety of reinforcement learning challenges. With its focus on ease of setup and comprehensive documentation, Deep Neuroevolution is accessible to anyone looking to explore or advance in AI and deep learning fields.