laser

Layer-Selective Rank Reduction

The Layer-Selective Rank Reduction project, also known as LASER, is an innovative approach designed to enhance reasoning in Language Models (LLMs). This project was developed by a team of researchers, including Pratyusha Sharma, Jordan T. Ash, and Dipendra Misra, and further details can be found in their paper submitted to ICLR 2024.

Project Overview

LASER aims to improve language models by introducing a technique that selectively reduces the rank of certain components within the model's architecture. Specifically, it replaces a chosen weight matrix in the transformer's architecture with a low-rank approximation, helping enhance the model's performance without requiring additional training. The method employs Singular Value Decomposition (SVD) for this reduction.

The technique is identified by three parameters:

1. Layer Number (ℓ): The specific layer to be modified, such as the 16th layer in the model.
2. Parameter Type (τ): The type of the matrix component, like the first MLP (Multi-Layer Perceptron) layer.
3. Rank Fraction (ρ): The fraction of the matrix's rank to be retained, for instance, keeping 1% of the rank.

Installation and Execution

To try out LASER, users can install the necessary components via a pip file, requiring PyTorch and libraries from Hugging Face. It's recommended to set up a conda environment to manage these dependencies.

pip3 install -r requirements.txt

For experimenting, especially with the CounterFact dataset, the following script is used to download it:

python scripts/get_counterfact.py

A simple experiment can be executed by modifying a specific transformer in the GPTJ model based on the Fever dataset:

python3 intervention_gptj_fever.py --lname fc_in --rate 9.9 --lnum 26

This command adapts the parameters τ and ρ to perform the LASER intervention.

Code Structure

The project’s code is organized under the src directory. Each experiment is wrapped in a Python script named by the pattern intervention_<llm-name>_<dataset-name>.py. The transformations and low-rank operations are facilitated by custom matrix utilities coded within the project.

Additionally, users are encouraged to customize and adapt the LASER technique to new language models by updating relevant mappings and wrappers in the code.

Contribution and Citation

Contributors can report new results by making pull requests, which will then be added to the project's leaderboard. The project team is also open to suggestions and discussions on improving LASER's capabilities and flexibility.

For those utilizing the LASER codebase in their work, the project authors request acknowledgment by citing their paper as follows:

@article{sharma2023truth,
    title={The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction},
    author={Sharma, Pratyusha and Ash, Jordan T and Misra, Dipendra},
    journal={arXiv preprint arXiv:2312.13558},
    year={2023}
}

Conclusion

LASER presents a promising methodology to enhance the efficiency and capability of language models through strategic structural tweaks. By embracing these innovative adjustments, users can achieve notable improvements in model performance, particularly on question-answering tasks, while maintaining a lean computational overhead.