BERT(S) for Relation Extraction
Overview
The BERT-Relation-Extraction project involves implementing models based on a significant paper titled "Matching the Blanks: Distributional Similarity for Relation Learning", presented at ACL 2019. This project hinges on PyTorch and embraces various models, including ALBERT and BioBERT, designed for extracting relationships from text data. Although it's not an official repository of the paper, it follows its methodology to explore new dimensions of relation extraction.
For further conceptual understanding of this implementation, a comprehensive guide can be found here.
Requirements
To make the most out of this project, one needs to ensure the following setup:
- Python version 3.8 or higher
- Required Python libraries can be installed using this command:
python3 -m pip install -r requirements.txt - Additionally, the language model can be downloaded using:
python3 -m spacy download en_core_web_lg
Pre-trained models like ALBERT and BERT are available from HuggingFace.co, and the BioBERT model can be downloaded from DMIS Lab.
Training by Matching the Blanks
In the BERT-Relation-Extraction project, training is done through a process called Matching the Blanks (MTB). It begins with running a script called main_pretraining.py, which requires specific arguments to tailor the training:
- The pre-training data can be any simple text file, with functionality powered by Spacy NLP to extract pairwise entities.
- While MTB pre-training can be lengthy, especially without a strong GPU, one may proceed directly to fine-tuning for adequate results.
The setup involves using CNN datasets for data processing, but larger datasets like Wiki dumps are recommended for MTB pre-training.
Fine-tuning on SemEval2010 Task 8
Fine-tuning leverages SemEval2010 Task 8 dataset using the main_task.py script, which can be downloaded through a linked source. This stage focuses on refining the model for more specific relation classification tasks.
Inference
The project allows for inferences where users can input sentences with identified entities using tags [E1] and [E2]. Here's a simple example:
Input:
The surprise [E1]visit[/E1] caused a [E2]frenzy[/E2] on the already chaotic trading floor.
The output will predict the type of relationship, such as:
Predicted: Cause-Effect(e1,e2)
The tool can also automatically detect relations between entities it identifies in a text, providing varied predictions based on detected combinations.
FewRel Task
This project also includes provisions for working on FewRel 1.0, a dataset aimed at relation classification. Users can run tasks tailored to FewRel by setting the argument in a particular way within the main_task.py script.
Benchmark Results
For the SemEval2010 Task 8, various configurations of models like BERT and ALBERT were tested to determine efficiency and accuracy. Results indicate satisfactory F1 scores, a measure of a model's accuracy, particularly when leverage training data fully.
To-Do
The project expects future extensions, which will include inference benchmarks and extending functionality for the FewRel task. This provides opportunities for continual improvements and refining existing methodologies further.
This comprehensive walkthrough provides a clear understanding of what the BERT-Relation-Extraction project encapsulates, guiding enthusiasts and professionals in the challenging yet fascinating domain of relation extraction.