pytorch-openai-transformer-lm

Introduction to PyTorch-OpenAI-Transformer Language Model

The PyTorch implementation of OpenAI's Finetuned Transformer Language Model is a translation of the original TensorFlow-based code developed by Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. This project aims to bring the powerful capabilities of OpenAI's pre-trained transformer language models into the PyTorch ecosystem. Here's a comprehensive overview of what this project entails.

Background

OpenAI's paper, "Improving Language Understanding by Generative Pre-Training," introduces a groundbreaking approach to enhancing language models through generative pre-training. This PyTorch project mirrors the TensorFlow implementation to bring similar results in language understanding tasks using PyTorch's flexibility and dynamic computation graph support.

Overview of the Model

At its core, the project provides a PyTorch implementation of OpenAI's transformer language model. It includes a script designed to incorporate pre-trained weights, ensuring compatibility with the original TensorFlow network structure.

The repository contains:

• model_pytorch.py: This file includes the model classes and the loading script. The module and variable names align closely with those in the TensorFlow version to ensure seamless conversion.
• Modified Adam Optimizer: The project utilizes an adjusted Adam optimization algorithm for training, incorporating modifications such as fixed weight decay and a scheduled learning rate to optimize performance for transformer models.

Requirements

To utilize the PyTorch model (model_pytorch.py), the main requirement is PyTorch (version 0.4 or higher). For running the classifier training script (train.py), additional dependencies include tqdm, sklearn, spacy, ftfy, and pandas.

To get started with the pre-trained weights, one needs to clone Alec Radford's repository and place the model folder containing the weights into the current repository.

Using the Pre-Trained Model

Utilizing the OpenAI pre-trained weights as a transformer language model is straightforward. Here's a basic code snippet to load and use the model:

from model_pytorch import TransformerModel, load_openai_pretrained_model, DEFAULT_CONFIG

args = DEFAULT_CONFIG
model = TransformerModel(args)
load_openai_pretrained_model(model)

This setup generates the Transformer's hidden states, which can be further used for tasks like language modeling or classification. The project includes classes like LMHead for building a complete language model and ClfHead for establishing a classifier as explained in OpenAI's publication.

Fine-Tuning on Classification Tasks

The model is not limited to general language tasks; it can be fine-tuned for specific tasks, such as the ROCStories Cloze task, a classification challenge highlighted in OpenAI's paper. Instructions and the necessary training code for fine-tuning are found in train.py, and the dataset can be downloaded from the specified website.

Fine-tuning involves just a few straightforward steps:

1. Download necessary NLP tools (like spaCy).
2. Prepare the dataset.
3. Execute train.py with relevant parameters.

Initial Experiments and Performance

Initial experiments demonstrate that fine-tuning the model for 3 Epochs on the ROCStories task takes about 10 minutes on an NVidia K-80 GPU. It achieves a test accuracy of 85.84%, closely mirroring results obtained with the original TensorFlow model and even surpassing past benchmarks on the ROCStories dataset, which were recorded at 77.6%.

Conclusion

This PyTorch implementation opens opportunities for developers and researchers to leverage OpenAI's state-of-the-art language models within a familiar and flexible framework. By following the structure and methodology laid out by OpenAI, including model weights and optimization strategies, this project enables continued advancements in language understanding tasks using PyTorch.